From patchwork Wed Feb 15 01:20:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 13141167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4BACC05027 for ; Wed, 15 Feb 2023 01:22:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232942AbjBOBWP (ORCPT ); Tue, 14 Feb 2023 20:22:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229923AbjBOBWO (ORCPT ); Tue, 14 Feb 2023 20:22:14 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DE973403D; Tue, 14 Feb 2023 17:21:37 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E0079B81F41; Wed, 15 Feb 2023 01:21:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ECF35C4339B; Wed, 15 Feb 2023 01:21:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676424093; bh=NRW+95kklKn9VOfFLptRsV2/h0nXYJrNW+qc67RxyAk=; h=From:To:Cc:Subject:Date:From; b=QmkkmFspZR/it2qxNF9/cVq+LPeoZMpETLuQ3VfNjupnpHJeZkQ3Jsqe0oDmSx+QG ukKg3srdhD2OWS3bpKYvlTkQq8n52L9Dymu4yXdl5ibgi3GwEdXYHSAte6DFVFQCHF KSfA2Kbkv3HlfqMyZOBj80hVtToSkkavJMwYBDNUjxWNHpUKgyYlLGCuR9sr6nj2g0 Q7Pshp698iLqDvXy3EI1ZF0qjgBsoP8gT/W85ta0B3x1jLlhdS99vvxr55oIpXzkm+ o3JwwH5YtzhRsPjGBaP1FmS8rjXUV3xIpziAUaTIy70a2hR23tPRBbzqcoJJZjfFrB juB+iA4D6FhFw== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ben Hutchings , Masahiro Yamada , Nathan Chancellor , Nick Desaulniers , Nicolas Schier Subject: [PATCH v6 01/12] kbuild: add a tool to list files ignored by git Date: Wed, 15 Feb 2023 10:20:23 +0900 Message-Id: <20230215012034.403356-1-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org In short, the motivation of this commit is to build a source package without cleaning the source tree. The deb-pkg and (src)rpm-pkg targets first run 'make clean' before creating a source tarball. Otherwise build artifacts such as *.o, *.a, etc. would be included in the tarball. Yet, the tarball ends up containing several garbage files since 'make clean' does not clean everything. Cleaning the tree every time is annoying since it makes the incremental build impossible. It is desirable to create a source tarball without cleaning the tree. In fact, there are some ways to achieve this. The easiest solution is 'git archive'. 'make perf-tar*-src-pkg' uses it, but I do not like it because it works only when the source tree is managed by git, and all files you want in the tarball must be committed in advance. I want to make it work without relying on git. We can do this. Files that are ignored by git are generated files, so should be excluded from the source tarball. We can list them out by parsing the .gitignore files. Of course, .gitignore does not cover all the cases, but it works well enough. tar(1) claims to support it: --exclude-vcs-ignores Exclude files that match patterns read from VCS-specific ignore files. Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore. The best scenario would be to use 'tar --exclude-vcs-ignores', but this option does not work. --exclude-vcs-ignore does not understand any of the negation (!), preceding slash, following slash, etc.. So, this option is just useless. Hence, I wrote this gitignore parser. The previous version [1], written in Python, was so slow. This version is implemented in C, so it works much faster. I imported the code from git (commit: 23c56f7bd5f1), so we get the same result. This tool traverses the source tree, parsing all .gitignore files, and prints file paths that are ignored by git. The output is similar to 'git ls-files --ignored --directory --others --exclude-per-directory=.gitignore', except [1] Not sorted [2] No trailing slash for directories [2] is intentional because tar's --exclude-from option cannot handle trailing slashes. [How to test this tool] $ git clean -dfx $ make -s -j$(nproc) defconfig all # or allmodconifg or whatever $ git archive -o ../linux1.tar --prefix=./ HEAD $ tar tf ../linux1.tar | LANG=C sort > ../file-list1 # files emitted by 'git archive' $ make scripts_package HOSTCC scripts/list-gitignored $ scripts/list-gitignored --prefix=./ -o ../exclude-list $ tar cf ../linux2.tar --exclude-from=../exclude-list . $ tar tf ../linux2.tar | LANG=C sort > ../file-list2 # files emitted by 'tar' $ diff ../file-list1 ../file-list2 | grep -E '^(<|>)' < ./Documentation/devicetree/bindings/.yamllint < ./drivers/clk/.kunitconfig < ./drivers/gpu/drm/tests/.kunitconfig < ./drivers/hid/.kunitconfig < ./fs/ext4/.kunitconfig < ./fs/fat/.kunitconfig < ./kernel/kcsan/.kunitconfig < ./lib/kunit/.kunitconfig < ./mm/kfence/.kunitconfig < ./tools/testing/selftests/arm64/tags/ < ./tools/testing/selftests/arm64/tags/.gitignore < ./tools/testing/selftests/arm64/tags/Makefile < ./tools/testing/selftests/arm64/tags/run_tags_test.sh < ./tools/testing/selftests/arm64/tags/tags_test.c < ./tools/testing/selftests/kvm/.gitignore < ./tools/testing/selftests/kvm/Makefile < ./tools/testing/selftests/kvm/config < ./tools/testing/selftests/kvm/settings The source tarball contains most of files that are tracked by git. You see some diffs, but it is just because some .gitignore files are wrong. $ git ls-files -i -c --exclude-per-directory=.gitignore Documentation/devicetree/bindings/.yamllint drivers/clk/.kunitconfig drivers/gpu/drm/tests/.kunitconfig drivers/hid/.kunitconfig fs/ext4/.kunitconfig fs/fat/.kunitconfig kernel/kcsan/.kunitconfig lib/kunit/.kunitconfig mm/kfence/.kunitconfig tools/testing/selftests/arm64/tags/.gitignore tools/testing/selftests/arm64/tags/Makefile tools/testing/selftests/arm64/tags/run_tags_test.sh tools/testing/selftests/arm64/tags/tags_test.c tools/testing/selftests/kvm/.gitignore tools/testing/selftests/kvm/Makefile tools/testing/selftests/kvm/config tools/testing/selftests/kvm/settings [1]: https://lore.kernel.org/all/20230128173843.765212-1-masahiroy@kernel.org/ Signed-off-by: Masahiro Yamada --- (no changes since v5) Changes in v5: - Import a lot of code from GIT - Fix a bug that the output file is relative to --rootdir - Check ENOENT when open() fails to open .gitignore - add more options, --ignore-case, --stat Changes in v3: - Various code refactoring: remove struct gitignore, remove next: label etc. - Support --extra-pattern option Changes in v2: - Reimplement in C Makefile | 7 +- scripts/.gitignore | 1 + scripts/Makefile | 2 +- scripts/list-gitignored.c | 1055 +++++++++++++++++++++++++++++++++++++ 4 files changed, 1063 insertions(+), 2 deletions(-) create mode 100644 scripts/list-gitignored.c diff --git a/Makefile b/Makefile index 2faf872b6808..7c726fd26540 100644 --- a/Makefile +++ b/Makefile @@ -274,7 +274,8 @@ no-dot-config-targets := $(clean-targets) \ cscope gtags TAGS tags help% %docs check% coccicheck \ $(version_h) headers headers_% archheaders archscripts \ %asm-generic kernelversion %src-pkg dt_binding_check \ - outputmakefile rustavailable rustfmt rustfmtcheck + outputmakefile rustavailable rustfmt rustfmtcheck \ + scripts_package # Installation targets should not require compiler. Unfortunately, vdso_install # is an exception where build artifacts may be updated. This must be fixed. no-compiler-targets := $(no-dot-config-targets) install dtbs_install \ @@ -1652,6 +1653,10 @@ distclean: mrproper %pkg: include/config/kernel.release FORCE $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.package $@ +PHONY += scripts_package +scripts_package: scripts_basic + $(Q)$(MAKE) $(build)=scripts scripts/list-gitignored + # Brief documentation of the typical targets used # --------------------------------------------------------------------------- diff --git a/scripts/.gitignore b/scripts/.gitignore index 6e9ce6720a05..feb43045d1b1 100644 --- a/scripts/.gitignore +++ b/scripts/.gitignore @@ -3,6 +3,7 @@ /generate_rust_target /insert-sys-cert /kallsyms +/list-gitignored /module.lds /recordmcount /sign-file diff --git a/scripts/Makefile b/scripts/Makefile index 32b6ba722728..e8917975905c 100644 --- a/scripts/Makefile +++ b/scripts/Makefile @@ -38,7 +38,7 @@ HOSTCFLAGS_sorttable.o += -DMCOUNT_SORT_ENABLED endif # The following programs are only built on demand -hostprogs += unifdef +hostprogs += list-gitignored unifdef # The module linker script is preprocessed on demand targets += module.lds diff --git a/scripts/list-gitignored.c b/scripts/list-gitignored.c new file mode 100644 index 000000000000..525c987c0e25 --- /dev/null +++ b/scripts/list-gitignored.c @@ -0,0 +1,1055 @@ +// SPDX-License-Identifier: GPL-2.0-only +// +// Traverse the source tree, parsing all .gitignore files, and print file paths +// that are ignored by git. +// The output is suitable to the --exclude-from option of tar. +// This is useful until the --exclude-vcs-ignores option gets working correctly. +// +// Copyright (C) 2023 Masahiro Yamada +// (a lot of code imported from GIT) + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +//---------------------------(IMPORT FROM GIT BEGIN)--------------------------- + +// Copied from environment.c + +static bool ignore_case; + +// Copied from git-compat-util.h + +/* Sane ctype - no locale, and works with signed chars */ +#undef isascii +#undef isspace +#undef isdigit +#undef isalpha +#undef isalnum +#undef isprint +#undef islower +#undef isupper +#undef tolower +#undef toupper +#undef iscntrl +#undef ispunct +#undef isxdigit + +static const unsigned char sane_ctype[256]; +#define GIT_SPACE 0x01 +#define GIT_DIGIT 0x02 +#define GIT_ALPHA 0x04 +#define GIT_GLOB_SPECIAL 0x08 +#define GIT_REGEX_SPECIAL 0x10 +#define GIT_PATHSPEC_MAGIC 0x20 +#define GIT_CNTRL 0x40 +#define GIT_PUNCT 0x80 +#define sane_istest(x,mask) ((sane_ctype[(unsigned char)(x)] & (mask)) != 0) +#define isascii(x) (((x) & ~0x7f) == 0) +#define isspace(x) sane_istest(x,GIT_SPACE) +#define isdigit(x) sane_istest(x,GIT_DIGIT) +#define isalpha(x) sane_istest(x,GIT_ALPHA) +#define isalnum(x) sane_istest(x,GIT_ALPHA | GIT_DIGIT) +#define isprint(x) ((x) >= 0x20 && (x) <= 0x7e) +#define islower(x) sane_iscase(x, 1) +#define isupper(x) sane_iscase(x, 0) +#define is_glob_special(x) sane_istest(x,GIT_GLOB_SPECIAL) +#define iscntrl(x) (sane_istest(x,GIT_CNTRL)) +#define ispunct(x) sane_istest(x, GIT_PUNCT | GIT_REGEX_SPECIAL | \ + GIT_GLOB_SPECIAL | GIT_PATHSPEC_MAGIC) +#define isxdigit(x) (hexval_table[(unsigned char)(x)] != -1) +#define tolower(x) sane_case((unsigned char)(x), 0x20) +#define toupper(x) sane_case((unsigned char)(x), 0) + +static inline int sane_case(int x, int high) +{ + if (sane_istest(x, GIT_ALPHA)) + x = (x & ~0x20) | high; + return x; +} + +static inline int sane_iscase(int x, int is_lower) +{ + if (!sane_istest(x, GIT_ALPHA)) + return 0; + + if (is_lower) + return (x & 0x20) != 0; + else + return (x & 0x20) == 0; +} + +// Copied from ctype.c + +enum { + S = GIT_SPACE, + A = GIT_ALPHA, + D = GIT_DIGIT, + G = GIT_GLOB_SPECIAL, /* *, ?, [, \\ */ + R = GIT_REGEX_SPECIAL, /* $, (, ), +, ., ^, {, | */ + P = GIT_PATHSPEC_MAGIC, /* other non-alnum, except for ] and } */ + X = GIT_CNTRL, + U = GIT_PUNCT, + Z = GIT_CNTRL | GIT_SPACE +}; + +static const unsigned char sane_ctype[256] = { + X, X, X, X, X, X, X, X, X, Z, Z, X, X, Z, X, X, /* 0.. 15 */ + X, X, X, X, X, X, X, X, X, X, X, X, X, X, X, X, /* 16.. 31 */ + S, P, P, P, R, P, P, P, R, R, G, R, P, P, R, P, /* 32.. 47 */ + D, D, D, D, D, D, D, D, D, D, P, P, P, P, P, G, /* 48.. 63 */ + P, A, A, A, A, A, A, A, A, A, A, A, A, A, A, A, /* 64.. 79 */ + A, A, A, A, A, A, A, A, A, A, A, G, G, U, R, P, /* 80.. 95 */ + P, A, A, A, A, A, A, A, A, A, A, A, A, A, A, A, /* 96..111 */ + A, A, A, A, A, A, A, A, A, A, A, R, R, U, P, X, /* 112..127 */ + /* Nothing in the 128.. range */ +}; + +// Copied from hex.c + +static const signed char hexval_table[256] = { + -1, -1, -1, -1, -1, -1, -1, -1, /* 00-07 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 08-0f */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 10-17 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 18-1f */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 20-27 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 28-2f */ + 0, 1, 2, 3, 4, 5, 6, 7, /* 30-37 */ + 8, 9, -1, -1, -1, -1, -1, -1, /* 38-3f */ + -1, 10, 11, 12, 13, 14, 15, -1, /* 40-47 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 48-4f */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 50-57 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 58-5f */ + -1, 10, 11, 12, 13, 14, 15, -1, /* 60-67 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 68-67 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 70-77 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 78-7f */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 80-87 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 88-8f */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 90-97 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 98-9f */ + -1, -1, -1, -1, -1, -1, -1, -1, /* a0-a7 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* a8-af */ + -1, -1, -1, -1, -1, -1, -1, -1, /* b0-b7 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* b8-bf */ + -1, -1, -1, -1, -1, -1, -1, -1, /* c0-c7 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* c8-cf */ + -1, -1, -1, -1, -1, -1, -1, -1, /* d0-d7 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* d8-df */ + -1, -1, -1, -1, -1, -1, -1, -1, /* e0-e7 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* e8-ef */ + -1, -1, -1, -1, -1, -1, -1, -1, /* f0-f7 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* f8-ff */ +}; + +// Copied from wildmatch.h + +#define WM_CASEFOLD 1 +#define WM_PATHNAME 2 + +#define WM_NOMATCH 1 +#define WM_MATCH 0 +#define WM_ABORT_ALL -1 +#define WM_ABORT_TO_STARSTAR -2 + +// Copied from wildmatch.c + +typedef unsigned char uchar; + +// local modification: remove NEGATE_CLASS(2) + +#define CC_EQ(class, len, litmatch) ((len) == sizeof (litmatch)-1 \ + && *(class) == *(litmatch) \ + && strncmp((char*)class, litmatch, len) == 0) + +// local modification: simpilify macros +#define ISBLANK(c) ((c) == ' ' || (c) == '\t') +#define ISGRAPH(c) (isprint(c) && !isspace(c)) +#define ISPRINT(c) isprint(c) +#define ISDIGIT(c) isdigit(c) +#define ISALNUM(c) isalnum(c) +#define ISALPHA(c) isalpha(c) +#define ISCNTRL(c) iscntrl(c) +#define ISLOWER(c) islower(c) +#define ISPUNCT(c) ispunct(c) +#define ISSPACE(c) isspace(c) +#define ISUPPER(c) isupper(c) +#define ISXDIGIT(c) isxdigit(c) + +/* Match pattern "p" against "text" */ +static int dowild(const uchar *p, const uchar *text, unsigned int flags) +{ + uchar p_ch; + const uchar *pattern = p; + + for ( ; (p_ch = *p) != '\0'; text++, p++) { + int matched, match_slash, negated; + uchar t_ch, prev_ch; + if ((t_ch = *text) == '\0' && p_ch != '*') + return WM_ABORT_ALL; + if ((flags & WM_CASEFOLD) && ISUPPER(t_ch)) + t_ch = tolower(t_ch); + if ((flags & WM_CASEFOLD) && ISUPPER(p_ch)) + p_ch = tolower(p_ch); + switch (p_ch) { + case '\\': + /* Literal match with following character. Note that the test + * in "default" handles the p[1] == '\0' failure case. */ + p_ch = *++p; + /* FALLTHROUGH */ + default: + if (t_ch != p_ch) + return WM_NOMATCH; + continue; + case '?': + /* Match anything but '/'. */ + if ((flags & WM_PATHNAME) && t_ch == '/') + return WM_NOMATCH; + continue; + case '*': + if (*++p == '*') { + const uchar *prev_p = p - 2; + while (*++p == '*') {} + if (!(flags & WM_PATHNAME)) + /* without WM_PATHNAME, '*' == '**' */ + match_slash = 1; + else if ((prev_p < pattern || *prev_p == '/') && + (*p == '\0' || *p == '/' || + (p[0] == '\\' && p[1] == '/'))) { + /* + * Assuming we already match 'foo/' and are at + * , just assume it matches + * nothing and go ahead match the rest of the + * pattern with the remaining string. This + * helps make foo/<*><*>/bar (<> because + * otherwise it breaks C comment syntax) match + * both foo/bar and foo/a/bar. + */ + if (p[0] == '/' && + dowild(p + 1, text, flags) == WM_MATCH) + return WM_MATCH; + match_slash = 1; + } else /* WM_PATHNAME is set */ + match_slash = 0; + } else + /* without WM_PATHNAME, '*' == '**' */ + match_slash = flags & WM_PATHNAME ? 0 : 1; + if (*p == '\0') { + /* Trailing "**" matches everything. Trailing "*" matches + * only if there are no more slash characters. */ + if (!match_slash) { + if (strchr((char *)text, '/')) + return WM_NOMATCH; + } + return WM_MATCH; + } else if (!match_slash && *p == '/') { + /* + * _one_ asterisk followed by a slash + * with WM_PATHNAME matches the next + * directory + */ + const char *slash = strchr((char*)text, '/'); + if (!slash) + return WM_NOMATCH; + text = (const uchar*)slash; + /* the slash is consumed by the top-level for loop */ + break; + } + while (1) { + if (t_ch == '\0') + break; + /* + * Try to advance faster when an asterisk is + * followed by a literal. We know in this case + * that the string before the literal + * must belong to "*". + * If match_slash is false, do not look past + * the first slash as it cannot belong to '*'. + */ + if (!is_glob_special(*p)) { + p_ch = *p; + if ((flags & WM_CASEFOLD) && ISUPPER(p_ch)) + p_ch = tolower(p_ch); + while ((t_ch = *text) != '\0' && + (match_slash || t_ch != '/')) { + if ((flags & WM_CASEFOLD) && ISUPPER(t_ch)) + t_ch = tolower(t_ch); + if (t_ch == p_ch) + break; + text++; + } + if (t_ch != p_ch) + return WM_NOMATCH; + } + if ((matched = dowild(p, text, flags)) != WM_NOMATCH) { + if (!match_slash || matched != WM_ABORT_TO_STARSTAR) + return matched; + } else if (!match_slash && t_ch == '/') + return WM_ABORT_TO_STARSTAR; + t_ch = *++text; + } + return WM_ABORT_ALL; + case '[': + p_ch = *++p; + if (p_ch == '^') + p_ch = '!'; + /* Assign literal 1/0 because of "matched" comparison. */ + negated = p_ch == '!' ? 1 : 0; + if (negated) { + /* Inverted character class. */ + p_ch = *++p; + } + prev_ch = 0; + matched = 0; + do { + if (!p_ch) + return WM_ABORT_ALL; + if (p_ch == '\\') { + p_ch = *++p; + if (!p_ch) + return WM_ABORT_ALL; + if (t_ch == p_ch) + matched = 1; + } else if (p_ch == '-' && prev_ch && p[1] && p[1] != ']') { + p_ch = *++p; + if (p_ch == '\\') { + p_ch = *++p; + if (!p_ch) + return WM_ABORT_ALL; + } + if (t_ch <= p_ch && t_ch >= prev_ch) + matched = 1; + else if ((flags & WM_CASEFOLD) && ISLOWER(t_ch)) { + uchar t_ch_upper = toupper(t_ch); + if (t_ch_upper <= p_ch && t_ch_upper >= prev_ch) + matched = 1; + } + p_ch = 0; /* This makes "prev_ch" get set to 0. */ + } else if (p_ch == '[' && p[1] == ':') { + const uchar *s; + int i; + for (s = p += 2; (p_ch = *p) && p_ch != ']'; p++) {} /*SHARED ITERATOR*/ + if (!p_ch) + return WM_ABORT_ALL; + i = p - s - 1; + if (i < 0 || p[-1] != ':') { + /* Didn't find ":]", so treat like a normal set. */ + p = s - 2; + p_ch = '['; + if (t_ch == p_ch) + matched = 1; + continue; + } + if (CC_EQ(s,i, "alnum")) { + if (ISALNUM(t_ch)) + matched = 1; + } else if (CC_EQ(s,i, "alpha")) { + if (ISALPHA(t_ch)) + matched = 1; + } else if (CC_EQ(s,i, "blank")) { + if (ISBLANK(t_ch)) + matched = 1; + } else if (CC_EQ(s,i, "cntrl")) { + if (ISCNTRL(t_ch)) + matched = 1; + } else if (CC_EQ(s,i, "digit")) { + if (ISDIGIT(t_ch)) + matched = 1; + } else if (CC_EQ(s,i, "graph")) { + if (ISGRAPH(t_ch)) + matched = 1; + } else if (CC_EQ(s,i, "lower")) { + if (ISLOWER(t_ch)) + matched = 1; + } else if (CC_EQ(s,i, "print")) { + if (ISPRINT(t_ch)) + matched = 1; + } else if (CC_EQ(s,i, "punct")) { + if (ISPUNCT(t_ch)) + matched = 1; + } else if (CC_EQ(s,i, "space")) { + if (ISSPACE(t_ch)) + matched = 1; + } else if (CC_EQ(s,i, "upper")) { + if (ISUPPER(t_ch)) + matched = 1; + else if ((flags & WM_CASEFOLD) && ISLOWER(t_ch)) + matched = 1; + } else if (CC_EQ(s,i, "xdigit")) { + if (ISXDIGIT(t_ch)) + matched = 1; + } else /* malformed [:class:] string */ + return WM_ABORT_ALL; + p_ch = 0; /* This makes "prev_ch" get set to 0. */ + } else if (t_ch == p_ch) + matched = 1; + } while (prev_ch = p_ch, (p_ch = *++p) != ']'); + if (matched == negated || + ((flags & WM_PATHNAME) && t_ch == '/')) + return WM_NOMATCH; + continue; + } + } + + return *text ? WM_NOMATCH : WM_MATCH; +} + +/* Match the "pattern" against the "text" string. */ +static int wildmatch(const char *pattern, const char *text, unsigned int flags) +{ + // local modification: move WM_CASEFOLD here + if (ignore_case) + flags |= WM_CASEFOLD; + + return dowild((const uchar*)pattern, (const uchar*)text, flags); +} + +// copied from dir.h + +#define PATTERN_FLAG_NODIR 1 +#define PATTERN_FLAG_ENDSWITH 4 +#define PATTERN_FLAG_MUSTBEDIR 8 +#define PATTERN_FLAG_NEGATIVE 16 + +// Copied from dir.c + +static int fspathncmp(const char *a, const char *b, size_t count) +{ + return ignore_case ? strncasecmp(a, b, count) : strncmp(a, b, count); +} + +static int simple_length(const char *match) +{ + int len = -1; + + for (;;) { + unsigned char c = *match++; + len++; + if (c == '\0' || is_glob_special(c)) + return len; + } +} + +static int no_wildcard(const char *string) +{ + return string[simple_length(string)] == '\0'; +} + +static void parse_path_pattern(const char **pattern, + int *patternlen, + unsigned *flags, + int *nowildcardlen) +{ + const char *p = *pattern; + size_t i, len; + + *flags = 0; + if (*p == '!') { + *flags |= PATTERN_FLAG_NEGATIVE; + p++; + } + len = strlen(p); + if (len && p[len - 1] == '/') { + len--; + *flags |= PATTERN_FLAG_MUSTBEDIR; + } + for (i = 0; i < len; i++) { + if (p[i] == '/') + break; + } + if (i == len) + *flags |= PATTERN_FLAG_NODIR; + *nowildcardlen = simple_length(p); + /* + * we should have excluded the trailing slash from 'p' too, + * but that's one more allocation. Instead just make sure + * nowildcardlen does not exceed real patternlen + */ + if (*nowildcardlen > len) + *nowildcardlen = len; + if (*p == '*' && no_wildcard(p + 1)) + *flags |= PATTERN_FLAG_ENDSWITH; + *pattern = p; + *patternlen = len; +} + +static void trim_trailing_spaces(char *buf) +{ + char *p, *last_space = NULL; + + for (p = buf; *p; p++) + switch (*p) { + case ' ': + if (!last_space) + last_space = p; + break; + case '\\': + p++; + if (!*p) + return; + /* fallthrough */ + default: + last_space = NULL; + } + + if (last_space) + *last_space = '\0'; +} + +static int match_basename(const char *basename, int basenamelen, + const char *pattern, int prefix, int patternlen, + unsigned flags) +{ + if (prefix == patternlen) { + if (patternlen == basenamelen && + !fspathncmp(pattern, basename, basenamelen)) + return 1; + } else if (flags & PATTERN_FLAG_ENDSWITH) { + /* "*literal" matching against "fooliteral" */ + if (patternlen - 1 <= basenamelen && + !fspathncmp(pattern + 1, + basename + basenamelen - (patternlen - 1), + patternlen - 1)) + return 1; + } else { + // local modification: call wildmatch() directly + if (!wildmatch(pattern, basename, flags)) + return 1; + } + return 0; +} + +static int match_pathname(const char *pathname, int pathlen, + const char *base, int baselen, + const char *pattern, int prefix, int patternlen) +{ + // local modification: remove local variables + + /* + * match with FNM_PATHNAME; the pattern has base implicitly + * in front of it. + */ + if (*pattern == '/') { + pattern++; + patternlen--; + prefix--; + } + + /* + * baselen does not count the trailing slash. base[] may or + * may not end with a trailing slash though. + */ + if (pathlen < baselen + 1 || + (baselen && pathname[baselen] != '/') || + fspathncmp(pathname, base, baselen)) + return 0; + + // local modification: simplified because always baselen > 0 + pathname += baselen + 1; + pathlen -= baselen + 1; + + if (prefix) { + /* + * if the non-wildcard part is longer than the + * remaining pathname, surely it cannot match. + */ + if (prefix > pathlen) + return 0; + + if (fspathncmp(pattern, pathname, prefix)) + return 0; + pattern += prefix; + patternlen -= prefix; + pathname += prefix; + pathlen -= prefix; + + /* + * If the whole pattern did not have a wildcard, + * then our prefix match is all we need; we + * do not need to call fnmatch at all. + */ + if (!patternlen && !pathlen) + return 1; + } + + // local modification: call wildmatch() directly + return !wildmatch(pattern, pathname, WM_PATHNAME); +} + +// Copied from git/utf8.c + +static const char utf8_bom[] = "\357\273\277"; + +//----------------------------(IMPORT FROM GIT END)---------------------------- + +struct pattern { + unsigned int flags; + int nowildcardlen; + int patternlen; + int dirlen; + char pattern[]; +}; + +static struct pattern **pattern_list; +static int nr_patterns, alloced_patterns; + +// Remember the number of patterns at each directory level +static int *nr_patterns_at; +// Track the current/max directory level; +static int depth, max_depth; +static bool debug_on; +static FILE *out_fp, *stat_fp; +static char *prefix = ""; +static char *progname; + +static void __attribute__((noreturn)) perror_exit(const char *s) +{ + perror(s); + + exit(EXIT_FAILURE); +} + +static void __attribute__((noreturn)) error_exit(const char *fmt, ...) +{ + va_list args; + + fprintf(stderr, "%s: error: ", progname); + + va_start(args, fmt); + vfprintf(stderr, fmt, args); + va_end(args); + + exit(EXIT_FAILURE); +} + +static void debug(const char *fmt, ...) +{ + va_list args; + int i; + + if (!debug_on) + return; + + fprintf(stderr, "[DEBUG] "); + + for (i = 0; i < depth * 2; i++) + fputc(' ', stderr); + + va_start(args, fmt); + vfprintf(stderr, fmt, args); + va_end(args); +} + +static void *xrealloc(void *ptr, size_t size) +{ + ptr = realloc(ptr, size); + if (!ptr) + perror_exit(progname); + + return ptr; +} + +static void *xmalloc(size_t size) +{ + return xrealloc(NULL, size); +} + +// similar to last_matching_pattern_from_list() in GIT +static bool is_ignored(const char *path, int pathlen, int dirlen, bool is_dir) +{ + int i; + + // Search in the reverse order because the last matching pattern wins. + for (i = nr_patterns - 1; i >= 0; i--) { + struct pattern *p = pattern_list[i]; + unsigned int flags = p->flags; + const char *gitignore_dir = p->pattern + p->patternlen + 1; + bool ignored; + + if ((flags & PATTERN_FLAG_MUSTBEDIR) && !is_dir) + continue; + + if (flags & PATTERN_FLAG_NODIR) { + if (!match_basename(path + dirlen + 1, + pathlen - dirlen - 1, + p->pattern, + p->nowildcardlen, + p->patternlen, + p->flags)) + continue; + } else { + if (!match_pathname(path, pathlen, + gitignore_dir, p->dirlen, + p->pattern, + p->nowildcardlen, + p->patternlen)) + continue; + } + + debug("%s: matches %s%s%s (%s/.gitignore)\n", path, + flags & PATTERN_FLAG_NEGATIVE ? "!" : "", p->pattern, + flags & PATTERN_FLAG_MUSTBEDIR ? "/" : "", + gitignore_dir); + + ignored = (flags & PATTERN_FLAG_NEGATIVE) == 0; + if (ignored) + debug("Ignore: %s\n", path); + + return ignored; + } + + debug("%s: no match\n", path); + + return false; +} + +static void add_pattern(const char *string, const char *dir, int dirlen) +{ + struct pattern *p; + int patternlen, nowildcardlen; + unsigned int flags; + + parse_path_pattern(&string, &patternlen, &flags, &nowildcardlen); + + if (patternlen == 0) + return; + + p = xmalloc(sizeof(*p) + patternlen + dirlen + 2); + + memcpy(p->pattern, string, patternlen); + p->pattern[patternlen] = 0; + memcpy(p->pattern + patternlen + 1, dir, dirlen); + p->pattern[patternlen + 1 + dirlen] = 0; + + p->patternlen = patternlen; + p->nowildcardlen = nowildcardlen; + p->dirlen = dirlen; + p->flags = flags; + + debug("Add pattern: %s%s%s\n", + flags & PATTERN_FLAG_NEGATIVE ? "!" : "", p->pattern, + flags & PATTERN_FLAG_MUSTBEDIR ? "/" : ""); + + if (nr_patterns >= alloced_patterns) { + alloced_patterns += 128; + pattern_list = xrealloc(pattern_list, + sizeof(*pattern_list) * alloced_patterns); + } + + pattern_list[nr_patterns++] = p; +} + +// similar to add_patterns_from_buffer() in GIT +static void add_patterns_from_gitignore(const char *dir, int dirlen) +{ + struct stat st; + char path[PATH_MAX], *buf, *entry; + size_t size; + int fd, pathlen, i; + + pathlen = snprintf(path, sizeof(path), "%s/.gitignore", dir); + if (pathlen >= sizeof(path)) + error_exit("%s: too long path was truncated\n", path); + + fd = open(path, O_RDONLY | O_NOFOLLOW); + if (fd < 0) { + if (errno != ENOENT) + return perror_exit(path); + return; + } + + if (fstat(fd, &st) < 0) + perror_exit(path); + + size = st.st_size; + + buf = xmalloc(size + 1); + if (read(fd, buf, st.st_size) != st.st_size) + perror_exit(path); + + buf[st.st_size] = '\n'; + if (close(fd)) + perror_exit(path); + + debug("Parse %s\n", path); + + entry = buf; + + // skip utf8 bom + if (!strncmp(entry, utf8_bom, strlen(utf8_bom))) + entry += strlen(utf8_bom); + + for (i = entry - buf; i < size; i++) { + if (buf[i] == '\n') { + if (entry != buf + i && entry[0] != '#') { + buf[i - (i && buf[i-1] == '\r')] = 0; + trim_trailing_spaces(entry); + add_pattern(entry, dir, dirlen); + } + entry = buf + i + 1; + } + } + + free(buf); +} + +// Save the current number of patterns and increment the depth +static void increment_depth(void) +{ + if (depth >= max_depth) { + max_depth += 1; + nr_patterns_at = xrealloc(nr_patterns_at, + sizeof(*nr_patterns_at) * max_depth); + } + + nr_patterns_at[depth] = nr_patterns; + depth++; +} + +// Decrement the depth, and free up the patterns of this directory level. +static void decrement_depth(void) +{ + depth--; + assert(depth >= 0); + + while (nr_patterns > nr_patterns_at[depth]) + free(pattern_list[--nr_patterns]); +} + +static void print_path(const char *path) +{ + // The path always starts with "./" + assert(strlen(path) >= 2); + + // Replace the root directory with a preferred prefix. + // This is useful for the tar command. + fprintf(out_fp, "%s%s\n", prefix, path + 2); +} + +static void print_stat(const char *path, struct stat *st) +{ + if (!stat_fp) + return; + + if (!S_ISREG(st->st_mode) && !S_ISLNK(st->st_mode)) + return; + + assert(strlen(path) >= 2); + + fprintf(stat_fp, "%c %9ld %10ld %s\n", + S_ISLNK(st->st_mode) ? 'l' : '-', + st->st_size, st->st_mtim.tv_sec, path + 2); +} + +// Traverse the entire directory tree, parsing .gitignore files. +// Print file paths that are not tracked by git. +// +// Return true if all files under the directory are ignored, false otherwise. +static bool traverse_directory(const char *dir, int dirlen) +{ + bool all_ignored = true; + DIR *dirp; + + debug("Enter[%d]: %s\n", depth, dir); + increment_depth(); + + add_patterns_from_gitignore(dir, dirlen); + + dirp = opendir(dir); + if (!dirp) + perror_exit(dir); + + while (1) { + struct dirent *d; + struct stat st; + char path[PATH_MAX]; + int pathlen; + bool ignored; + + errno = 0; + d = readdir(dirp); + if (!d) { + if (errno) + perror_exit(dir); + break; + } + + if (!strcmp(d->d_name, "..") || !strcmp(d->d_name, ".")) + continue; + + pathlen = snprintf(path, sizeof(path), "%s/%s", dir, d->d_name); + if (pathlen >= sizeof(path)) + error_exit("%s: too long path was truncated\n", path); + + if (lstat(path, &st) < 0) + perror_exit(path); + + if ((!S_ISREG(st.st_mode) && !S_ISDIR(st.st_mode) && !S_ISLNK(st.st_mode)) || + is_ignored(path, pathlen, dirlen, S_ISDIR(st.st_mode))) { + ignored = true; + } else { + if (S_ISDIR(st.st_mode) && !S_ISLNK(st.st_mode)) + // If all the files in a directory are ignored, + // let's ignore that directory as well. This + // will avoid empty directories in the tarball. + ignored = traverse_directory(path, pathlen); + else + ignored = false; + } + + if (ignored) { + print_path(path); + } else { + print_stat(path, &st); + all_ignored = false; + } + } + + if (closedir(dirp)) + perror_exit(dir); + + decrement_depth(); + debug("Leave[%d]: %s\n", depth, dir); + + return all_ignored; +} + +static void usage(void) +{ + fprintf(stderr, + "usage: %s [options]\n" + "\n" + "Show files that are ignored by git\n" + "\n" + "options:\n" + " -d, --debug print debug messages to stderr\n" + " -e, --exclude PATTERN add the given exclude pattern\n" + " -h, --help show this help message and exit\n" + " -i, --ignore-case Ignore case differences between the patterns and the files\n" + " -o, --output FILE output the ignored files to a file (default: '-', i.e. stdout)\n" + " -p, --prefix PREFIX prefix added to each path (default: empty string)\n" + " -r, --rootdir DIR root of the source tree (default: current working directory)\n" + " -s, --stat FILE output the file stat of non-ignored files to a file\n", + progname); +} + +static void open_output(const char *pathname, FILE **fp) +{ + if (strcmp(pathname, "-")) { + *fp = fopen(pathname, "w"); + if (!*fp) + perror_exit(pathname); + } else { + *fp = stdout; + } +} + +static void close_output(const char *pathname, FILE *fp) +{ + fflush(fp); + + if (ferror(fp)) + error_exit("not all data was written to the output\n"); + + if (fclose(fp)) + perror_exit(pathname); +} + +int main(int argc, char *argv[]) +{ + const char *output = "-"; + const char *rootdir = "."; + const char *stat = NULL; + + progname = strrchr(argv[0], '/'); + if (progname) + progname++; + else + progname = argv[0]; + + while (1) { + static struct option long_options[] = { + {"debug", no_argument, NULL, 'd'}, + {"help", no_argument, NULL, 'h'}, + {"ignore-case", no_argument, NULL, 'i'}, + {"output", required_argument, NULL, 'o'}, + {"prefix", required_argument, NULL, 'p'}, + {"rootdir", required_argument, NULL, 'r'}, + {"stat", required_argument, NULL, 's'}, + {"exclude", required_argument, NULL, 'x'}, + {}, + }; + + int c = getopt_long(argc, argv, "dhino:p:r:s:x:", long_options, NULL); + + if (c == -1) + break; + + switch (c) { + case 'd': + debug_on = true; + break; + case 'h': + usage(); + exit(0); + case 'i': + ignore_case = true; + break; + case 'o': + output = optarg; + break; + case 'p': + prefix = optarg; + break; + case 'r': + rootdir = optarg; + break; + case 's': + stat = optarg; + break; + case 'x': + add_pattern(optarg, ".", strlen(".")); + break; + case '?': + usage(); + /* fallthrough */ + default: + exit(EXIT_FAILURE); + } + } + + open_output(output, &out_fp); + if (stat && stat[0]) + open_output(stat, &stat_fp); + + if (chdir(rootdir)) + perror_exit(rootdir); + + add_pattern(".git/", ".", strlen(".")); + + if (traverse_directory(".", strlen("."))) + print_path("./"); + + assert(depth == 0); + + while (nr_patterns > 0) + free(pattern_list[--nr_patterns]); + free(pattern_list); + free(nr_patterns_at); + + close_output(output, out_fp); + if (stat_fp) + close_output(stat, stat_fp); + + return 0; +} From patchwork Wed Feb 15 01:20:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 13141169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36658C6379F for ; Wed, 15 Feb 2023 01:22:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233468AbjBOBWQ (ORCPT ); Tue, 14 Feb 2023 20:22:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43898 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232227AbjBOBWO (ORCPT ); Tue, 14 Feb 2023 20:22:14 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35B397EEC; Tue, 14 Feb 2023 17:21:39 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 8CB3ECE2317; Wed, 15 Feb 2023 01:21:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 338DCC4339E; Wed, 15 Feb 2023 01:21:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676424095; bh=x5ONHmD9axTWl2ROxj07e1ofChMPlgTyKLO4V4lNEIw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XTGt8ux85AQXhQI319C7K8yqIoGEXDYu9nX2yHvUA3kMCxc0xwn+2KBT/VSPeehpK imtbeyEfToMP/dxPY6opYBcjvGYuUcgyAiOgJE2Qm+9aSJGT3FR06vCdZJIehTNmXu Y+UgH2vcwlcc5LW1Gkj4wOgG5F9MjoCFUtP7QkNf1e3qT1WVA//GuvYkz2w7O/X8KK yKJy2C2ejSyz/eN5gc2k4lbqUy9ugcRuV7GtPr0+e+P0TLE7DJ8uOXZFhRfIz5JTsQ j06LtRkgdntpRlocZwLdAERa9iH2ATX2S5ykLABTxqJH9H6cGwR+jwpqQS1/2pE3Un y4/KdXMnIvfQQ== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ben Hutchings , Masahiro Yamada , Nathan Chancellor , Nick Desaulniers , Nicolas Schier , Tom Rix , llvm@lists.linux.dev Subject: [PATCH v6 02/12] kbuild: deb-pkg: create source package without cleaning Date: Wed, 15 Feb 2023 10:20:24 +0900 Message-Id: <20230215012034.403356-2-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230215012034.403356-1-masahiroy@kernel.org> References: <20230215012034.403356-1-masahiroy@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org If you run 'make deb-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible. Instead of cleaning, pass the exclude list to tar's --exclude-from option. Previously, *.diff.gz contained some check-in files such as .clang-format, .cocciconfig. With this commit, *.diff.gz will only contain the .config and debian/. The other source files will go into the .orig tarball. linux.tar.gz is rebuilt only when the source files that would go into the tarball are changed. Signed-off-by: Masahiro Yamada Reviewed-by: Nicolas Schier --- Changes in v6: - Add more stubs to cmd_tar so I can reuse it for the other rules Changes in v5: - Avoid unneeded rebuild of the tarball when nothing in the source tree is changed. Changes in v4: - Fix a typo in comment Changes in v3: - Add --extra-pattern='*.rej' - Exclude symlinks at the toplevel - Add --sort=name tar option scripts/Makefile.package | 48 +++++++++++++++++++++++++++++++++++----- scripts/package/mkdebian | 24 ++++++++++++++++++++ 2 files changed, 66 insertions(+), 6 deletions(-) diff --git a/scripts/Makefile.package b/scripts/Makefile.package index dfbf40454a99..f0002ace4156 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -43,13 +43,47 @@ if test "$(objtree)" != "$(srctree)"; then \ echo >&2 " ERROR:"; \ echo >&2 " Building source tarball is not possible outside the"; \ echo >&2 " kernel source tree. Don't set KBUILD_OUTPUT, or use the"; \ - echo >&2 " binrpm-pkg or bindeb-pkg target instead."; \ + echo >&2 " binrpm-pkg target instead."; \ echo >&2; \ false; \ fi ; \ tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \ --transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3) +# .tmp_filelist .tmp_filelist_exclude +# --------------------------------------------------------------------------- + +scripts/list-gitignored: FORCE + $(Q)$(MAKE) -f $(srctree)/Makefile scripts_package + +# 1f5d3a6b6532e25a5cdf1f311956b2b03d343a48 removed '*.rej' from .gitignore, +# but it is definitely a generated file. +filechk_filelist = \ + $< --exclude='*.rej' --output=$@_exclude --prefix=./ --rootdir=$(srctree) --stat=- + +.tmp_filelist: scripts/list-gitignored FORCE + $(call filechk,filelist) + +# tarball +# --------------------------------------------------------------------------- + +quiet_cmd_tar = TAR $@ + cmd_tar = tar -c -f $@ $(tar-compress-opt) $(tar-exclude-opt) \ + --owner=0 --group=0 --sort=name \ + --transform 's:^\.:$*:S' -C $(tar-rootdir) . + +tar-rootdir := $(srctree) + +%.tar.gz: private tar-compress-opt := -I $(KGZIP) +%.tar.gz: + $(call cmd,tar) + +# Linux source tarball +# --------------------------------------------------------------------------- + +linux.tar.gz: tar-exclude-opt = --exclude=./$@ --exclude-from=$<_exclude +linux.tar.gz: .tmp_filelist + # rpm-pkg # --------------------------------------------------------------------------- PHONY += rpm-pkg @@ -80,13 +114,12 @@ binrpm-pkg: $(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec PHONY += deb-pkg -deb-pkg: - $(MAKE) clean +deb-pkg: linux.tar.gz $(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian - $(call cmd,src_tar,$(KDEB_SOURCENAME)) origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\ - mv $(KDEB_SOURCENAME).tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz - +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) --source-option=-sP -i.git -us -uc + cp linux.tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz + +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ + --build=source,binary --source-option=-sP -nc -us -uc PHONY += bindeb-pkg bindeb-pkg: @@ -174,4 +207,7 @@ help: @echo ' perf-tarxz-src-pkg - Build $(perf-tar).tar.xz source tarball' @echo ' perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball' +PHONY += FORCE +FORCE: + .PHONY: $(PHONY) diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian index c3bbef7a6754..68b13ef590ba 100755 --- a/scripts/package/mkdebian +++ b/scripts/package/mkdebian @@ -84,6 +84,8 @@ set_debarch() { fi } +rm -rf debian + # Some variables and settings used throughout the script version=$KERNELRELEASE if [ -n "$KDEB_PKGVERSION" ]; then @@ -135,6 +137,28 @@ fi mkdir -p debian/source/ echo "1.0" > debian/source/format +# Ugly: ignore anything except .config or debian/ +cat<<'EOF' > debian/source/local-options +diff-ignore + +extend-diff-ignore = ^[^.d] + +extend-diff-ignore = ^\.[^c] +extend-diff-ignore = ^\.c($|[^o]) +extend-diff-ignore = ^\.co($|[^n]) +extend-diff-ignore = ^\.con($|[^f]) +extend-diff-ignore = ^\.conf($|[^i]) +extend-diff-ignore = ^\.confi($|[^g]) +extend-diff-ignore = ^\.config. + +extend-diff-ignore = ^d($|[^e]) +extend-diff-ignore = ^de($|[^b]) +extend-diff-ignore = ^deb($|[^i]) +extend-diff-ignore = ^debi($|[^a]) +extend-diff-ignore = ^debia($|[^n]) +extend-diff-ignore = ^debian[^/] +EOF + echo $debarch > debian/arch extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)" extra_build_depends="$extra_build_depends, $(if_enabled_echo CONFIG_SYSTEM_TRUSTED_KEYRING libssl-dev:native)" From patchwork Wed Feb 15 01:20:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 13141168 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2752C64EC7 for ; Wed, 15 Feb 2023 01:22:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233444AbjBOBWP (ORCPT ); Tue, 14 Feb 2023 20:22:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232736AbjBOBWO (ORCPT ); Tue, 14 Feb 2023 20:22:14 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30B6D31E04; Tue, 14 Feb 2023 17:21:40 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D9352B81F8D; Wed, 15 Feb 2023 01:21:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2451FC4339C; Wed, 15 Feb 2023 01:21:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676424097; bh=79SOz62YcHFzQBbIrRtp7ktn5c/Q1u91mdZ/FPKD8Es=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QdK1MqD0cy4vSU00oKAcdUwnOigcJRqoysMfSGsMEb80YwZDevrNx08EQojR1pcgN MFd5H4jbETnqVx1Ao9ZIRbfxkMEJKF0Qpp9B8FcfYsZ3TAuE4BSXdKJL8NP2R2Th6o VrA863CE+fMDSsV0Xv3d3dYuLYyZMRQHh7zv6xmn7lOp6dPYa1FvbYSN5unwnDeJNC 6j4g6MA6MVeYVq3RuDg7PdrAhM7H8kotidhRKrw/5hd0rsu/cJjLCvdoak/gemCEbH c6cHgA/BqWO7CQBmmHlcCFfgnNXmx3splzUkCPDTkZc9frOPdJOHOVzYmiSo+cr6qP UxIkupwURKV/g== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ben Hutchings , Masahiro Yamada , Nathan Chancellor , Nick Desaulniers , Nicolas Schier Subject: [PATCH v6 03/12] kbuild: rpm-pkg: build binary packages from source rpm Date: Wed, 15 Feb 2023 10:20:25 +0900 Message-Id: <20230215012034.403356-3-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230215012034.403356-1-masahiroy@kernel.org> References: <20230215012034.403356-1-masahiroy@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org The build rules of rpm-pkg and srcrpm-pkg are almost the same. Remove the code duplication. Change rpm-pkg to build binary packages from the source package generated by srcrpm-pkg. This changes the output directory of the srpm generated by 'make rpm-pkg' because srcrpm-pkg overrides _srcrpmdir. Signed-off-by: Masahiro Yamada --- (no changes since v3) Changes in v3: - Explain that the source package location will be changed. scripts/Makefile.package | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/scripts/Makefile.package b/scripts/Makefile.package index f0002ace4156..55441d776d70 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -87,11 +87,9 @@ linux.tar.gz: .tmp_filelist # rpm-pkg # --------------------------------------------------------------------------- PHONY += rpm-pkg -rpm-pkg: - $(MAKE) clean - $(CONFIG_SHELL) $(MKSPEC) >$(objtree)/kernel.spec - $(call cmd,src_tar,$(KERNELPATH),kernel.spec) - +rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -ta $(KERNELPATH).tar.gz \ +rpm-pkg: srpm = $(shell rpmspec --srpm --query --queryformat='%{name}-%{VERSION}-%{RELEASE}.src.rpm' kernel.spec) +rpm-pkg: srcrpm-pkg + +rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -rb $(srpm) \ --define='_smp_mflags %{nil}' # srcrpm-pkg From patchwork Wed Feb 15 01:20:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 13141171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41873C64ED8 for ; Wed, 15 Feb 2023 01:22:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233479AbjBOBWR (ORCPT ); Tue, 14 Feb 2023 20:22:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233452AbjBOBWQ (ORCPT ); Tue, 14 Feb 2023 20:22:16 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A27AB59FD; Tue, 14 Feb 2023 17:21:42 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id EDC45CE2312; Wed, 15 Feb 2023 01:21:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 00AD3C433EF; Wed, 15 Feb 2023 01:21:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676424099; bh=cUer3+ZFLO/LgmrihaytqwQfyiyEi5va0uACiX5PzjU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pKCvBNDzdJuWbdfhIqMA0XaTFxBw75yUiceXlGQdN4nX+Tar7kmNMzwR22pNKTIrh sqlbDG+agka93SgGyHEd+pRp5r1nclicHfSmdmXchqV7amvGcwdrVomEh8BqCFtNgF jsyZQls4aTHqw80cogDmiKTu9tniw9AJ+3yKF7WqalsKOX32fl6X2BweSbJblnFBMP 6EEBd3dRQeDIrLntX9dAxLC7fU9GaJi4ilECoY+DjnZJKxFG4rinVCIRb7/jNZSapL COsCOKgEY7MudtlyUxSwQdPALGKujjUam4KK+wbJu80Ssl2QWfzSm8ZtBPE4EBbVF3 hnnAad9iInB9A== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ben Hutchings , Masahiro Yamada , Nathan Chancellor , Nick Desaulniers , Nicolas Schier Subject: [PATCH v6 04/12] kbuild: srcrpm-pkg: create source package without cleaning Date: Wed, 15 Feb 2023 10:20:26 +0900 Message-Id: <20230215012034.403356-4-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230215012034.403356-1-masahiroy@kernel.org> References: <20230215012034.403356-1-masahiroy@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org If you run 'make (src)rpm-pkg', all objects are lost due to 'make clean', which makes the incremental builds impossible. Instead of cleaning, pass the exclude list to tar's --exclude-from option. Previously, the .config was contained in the source tarball. With this commit, the source rpm consists of separate linux.tar.gz and .config. Remove stale comments. Now, 'make (src)rpm-pkg' works with O= option. Signed-off-by: Masahiro Yamada --- (no changes since v4) Changes in v4: - Do not delete the old tar command because it is still used by snap-pkg although snap-pkg is broken, and it does not work at all. scripts/Makefile.package | 32 ++++---------------------------- scripts/package/mkspec | 8 ++++---- 2 files changed, 8 insertions(+), 32 deletions(-) diff --git a/scripts/Makefile.package b/scripts/Makefile.package index 55441d776d70..7266a11855f4 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -3,27 +3,6 @@ include $(srctree)/scripts/Kbuild.include -# RPM target -# --------------------------------------------------------------------------- -# The rpm target generates two rpm files: -# /usr/src/packages/SRPMS/kernel-2.6.7rc2-1.src.rpm -# /usr/src/packages/RPMS/i386/kernel-2.6.7rc2-1..rpm -# The src.rpm files includes all source for the kernel being built -# The .rpm includes kernel configuration, modules etc. -# -# Process to create the rpm files -# a) clean the kernel -# b) Generate .spec file -# c) Build a tar ball, using symlink to make kernel version -# first entry in the path -# d) and pack the result to a tar.gz file -# e) generate the rpm files, based on kernel.spec -# - Use /. to avoid tar packing just the symlink - -# Note that the rpm-pkg target cannot be used with KBUILD_OUTPUT, -# but the binrpm-pkg target can; for some reason O= gets ignored. - -# Remove hyphens since they have special meaning in RPM filenames KERNELPATH := kernel-$(subst -,_,$(KERNELRELEASE)) KDEB_SOURCENAME ?= linux-upstream KBUILD_PKG_ROOTCMD ?="fakeroot -u" @@ -42,8 +21,7 @@ if test "$(objtree)" != "$(srctree)"; then \ echo >&2; \ echo >&2 " ERROR:"; \ echo >&2 " Building source tarball is not possible outside the"; \ - echo >&2 " kernel source tree. Don't set KBUILD_OUTPUT, or use the"; \ - echo >&2 " binrpm-pkg target instead."; \ + echo >&2 " kernel source tree. Don't set KBUILD_OUTPUT"; \ echo >&2; \ false; \ fi ; \ @@ -95,12 +73,10 @@ rpm-pkg: srcrpm-pkg # srcrpm-pkg # --------------------------------------------------------------------------- PHONY += srcrpm-pkg -srcrpm-pkg: - $(MAKE) clean +srcrpm-pkg: linux.tar.gz $(CONFIG_SHELL) $(MKSPEC) >$(objtree)/kernel.spec - $(call cmd,src_tar,$(KERNELPATH),kernel.spec) - +rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -ts $(KERNELPATH).tar.gz \ - --define='_smp_mflags %{nil}' --define='_srcrpmdir $(srctree)' + +rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -bs kernel.spec \ + --define='_smp_mflags %{nil}' --define='_sourcedir .' --define='_srcrpmdir .' # binrpm-pkg # --------------------------------------------------------------------------- diff --git a/scripts/package/mkspec b/scripts/package/mkspec index 108c0cb95436..83a64d9d7372 100755 --- a/scripts/package/mkspec +++ b/scripts/package/mkspec @@ -47,7 +47,8 @@ sed -e '/^DEL/d' -e 's/^\t*//' < X-Patchwork-Id: 13141172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC287C677F1 for ; Wed, 15 Feb 2023 01:22:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233494AbjBOBWS (ORCPT ); Tue, 14 Feb 2023 20:22:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233472AbjBOBWQ (ORCPT ); Tue, 14 Feb 2023 20:22:16 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8DEDD34C05; Tue, 14 Feb 2023 17:21:43 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 3AC6AB81F80; Wed, 15 Feb 2023 01:21:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AEAECC433A1; Wed, 15 Feb 2023 01:21:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676424100; bh=m+DrF7xUfebZrPlWlJzQzOabnTDSzTMsjtSeJbQGQcQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=keJh3pqBunzYBbvrNdWRyQw12YA80yrXwQ117+xD4T6qVvvpLz/Pf2XK+Tkdz2Ffq 0uCdJVJc70YuYkXBbTbPiqwyH1k3RA3MzA1ZOPksjrvfRGxK0Wi5cPxq4s9XhlBk8V ku1FV4GXVVd0bjCxqSx2igYx+rp2/Ef8u90GFxLGqQs0NHtjZrl+EsmwzYpk8swzUb 1hbvKV/Bx1KKXC4HpPQry2KchR9n1KSisccvYjTpA/kmG09TF39HUvVoyceveODbfa RivfjwIIHepT3WB69+OREobV35CZ0VLkTcUhoYIFazXbNVJaP3cJleqVuxN6PTBKQi syNmxF57Wo40w== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ben Hutchings , Masahiro Yamada , Nathan Chancellor , Nick Desaulniers , Nicolas Schier Subject: [PATCH v6 05/12] kbuild: deb-pkg: hide KDEB_SOURCENAME from Makefile Date: Wed, 15 Feb 2023 10:20:27 +0900 Message-Id: <20230215012034.403356-5-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230215012034.403356-1-masahiroy@kernel.org> References: <20230215012034.403356-1-masahiroy@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org scripts/Makefile.package does not need to know the value of KDEB_SOURCENAME because the source name can be taken from debian/changelog by using dpkg-parsechangelog. Move the default of KDEB_SOURCENAME (i.e. linux-upstream) to scripts/package/mkdebian. Signed-off-by: Masahiro Yamada Reviewed-by: Nicolas Schier --- (no changes since v5) Changes in v5: - Copy linux.tar.gz to .orig tarball Changes in v3: - Move cmd_debianize Changes in v2: - New patch scripts/Makefile.package | 23 +++++++++++++++-------- scripts/package/mkdebian | 2 +- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/scripts/Makefile.package b/scripts/Makefile.package index 7266a11855f4..c207f4ec47ac 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -4,9 +4,7 @@ include $(srctree)/scripts/Kbuild.include KERNELPATH := kernel-$(subst -,_,$(KERNELRELEASE)) -KDEB_SOURCENAME ?= linux-upstream KBUILD_PKG_ROOTCMD ?="fakeroot -u" -export KDEB_SOURCENAME # Include only those top-level files that are needed by make, plus the GPL copy TAR_CONTENT := Documentation LICENSES arch block certs crypto drivers fs \ include init io_uring ipc kernel lib mm net rust \ @@ -87,17 +85,26 @@ binrpm-pkg: +rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \ $(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec +quiet_cmd_debianize = GEN $@ + cmd_debianize = $(srctree)/scripts/package/mkdebian + +debian: FORCE + $(call cmd,debianize) + +PHONY += debian-orig +debian-orig: private source = $(shell dpkg-parsechangelog -S Source) +debian-orig: private version = $(shell dpkg-parsechangelog -S Version | sed 's/-[^-]*$$//') +debian-orig: private orig-name = $(source)_$(version).orig.tar.gz +debian-orig: linux.tar.gz debian + $(Q)cp $< ../$(orig-name) + PHONY += deb-pkg -deb-pkg: linux.tar.gz - $(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian - origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\ - cp linux.tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz +deb-pkg: debian-orig +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ --build=source,binary --source-option=-sP -nc -us -uc PHONY += bindeb-pkg -bindeb-pkg: - $(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian +bindeb-pkg: debian +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc PHONY += intdeb-pkg diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian index 68b13ef590ba..534e9713761a 100755 --- a/scripts/package/mkdebian +++ b/scripts/package/mkdebian @@ -95,7 +95,7 @@ else revision=$($srctree/init/build-version) packageversion=$version-$revision fi -sourcename=$KDEB_SOURCENAME +sourcename=${KDEB_SOURCENAME:-linux-upstream} if [ "$ARCH" = "um" ] ; then packagename=user-mode-linux From patchwork Wed Feb 15 01:20:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 13141170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AADEC64ED6 for ; Wed, 15 Feb 2023 01:22:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233490AbjBOBWS (ORCPT ); Tue, 14 Feb 2023 20:22:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233478AbjBOBWQ (ORCPT ); Tue, 14 Feb 2023 20:22:16 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70B223250E; Tue, 14 Feb 2023 17:21:45 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 2379DB81F8C; Wed, 15 Feb 2023 01:21:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5CB81C433EF; Wed, 15 Feb 2023 01:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676424102; bh=nyoDlaWuMe+iAg2qQHK7dj0Pe2Isu5Lqca6xJRJDD9c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Y9fkk6E7z0wPKaR6y//S2W/6zS/FeQ4fIEcwplSHQMo8U7/VIXTc4XCeGaHC3YODt waoknC0wxfdDUnXW9FZmPWQMuB4BCb4NwkxBXRg1fiobnhpIhoxWCeEsagz1Jh+d5x ySyqpGMAHpyPaGaN9KHAlXSeXfz8cX+h+rXjwHjPps2y4cSfnRAkAXWZ8q0oqnIzh+ BpwIGJF9dn22ABNqG40VfGKO45Uv2im3Sij2Ws4ZusfrbZEb+mk0Ba8my/FezWRM7k NptGYmXD8PByRo3Q79fKJPu+rSIix4RajOJoVsIiyX0qauC/FnzSx5GSA3Dcko79bM nTVz4xzhteKKw== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ben Hutchings , Masahiro Yamada , Nathan Chancellor , Nick Desaulniers , Nicolas Schier Subject: [PATCH v6 06/12] kbuild: deb-pkg: make .orig tarball a hard link if possible Date: Wed, 15 Feb 2023 10:20:28 +0900 Message-Id: <20230215012034.403356-6-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230215012034.403356-1-masahiroy@kernel.org> References: <20230215012034.403356-1-masahiroy@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org If '..' belongs to the same filesystem, create a hard link instead of a copy. In most cases, you can save disk space. I do not want to use 'mv' because keeping linux.tar.gz is useful to avoid unneeded rebuilding of the tarball. Signed-off-by: Masahiro Yamada Reviewed-by: Nicolas Schier --- Changes in v6: - New patch scripts/Makefile.package | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/scripts/Makefile.package b/scripts/Makefile.package index c207f4ec47ac..80a96eb61426 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -96,7 +96,11 @@ debian-orig: private source = $(shell dpkg-parsechangelog -S Source) debian-orig: private version = $(shell dpkg-parsechangelog -S Version | sed 's/-[^-]*$$//') debian-orig: private orig-name = $(source)_$(version).orig.tar.gz debian-orig: linux.tar.gz debian - $(Q)cp $< ../$(orig-name) + $(Q)if [ "$(df --output=target .. 2>/dev/null)" = "$(df --output=target $< 2>/dev/null)" ]; then \ + ln -f $< ../$(orig-name); \ + else \ + cp $< ../$(orig-name); \ + fi PHONY += deb-pkg deb-pkg: debian-orig From patchwork Wed Feb 15 01:20:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 13141173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 409B3C61DA4 for ; Wed, 15 Feb 2023 01:22:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233486AbjBOBWV (ORCPT ); Tue, 14 Feb 2023 20:22:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44034 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233497AbjBOBWT (ORCPT ); Tue, 14 Feb 2023 20:22:19 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DB9334C08; Tue, 14 Feb 2023 17:21:47 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id EB762B81F8F; Wed, 15 Feb 2023 01:21:45 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4F6CCC433A0; Wed, 15 Feb 2023 01:21:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676424104; bh=1ifncx4mS+h9vwXHr2C/Xo2uXi8XsaHG4mWZBULRw1c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XtQoInXqwBBDqL5I5+ZdSX1wJAP66Af55RnkhqvPKRs2BxZyd4v56MJUN/onpbptt mp1flP19iQ/ANlIUuocu9BC1yPnSTiw/c6VkJPcIFn8LFRl2Du6SfXb8PujfxMVQ9T Sp6djTtBF39C9uHv1Gk2icPhAkugamAKnlZSDw8Afm4NqHwo76zgYq058yYuSiQNWr Uca0cLwmw4V2n+Eo/QJl0GqEsFoOxDnMx6ZrBh/8qMoSwRZaaSQSnJl8/SRa8PUp/u frMO7YWv4UabEq4FGtnIO6t57Rxut89k8WmdPDt7TdvvVglvs0ebH288PypFNVhVBy GP7KnJWdhBEnw== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ben Hutchings , Masahiro Yamada , Nathan Chancellor , Nick Desaulniers , Nicolas Schier Subject: [PATCH v6 07/12] kbuild: deb-pkg: switch over to source format 3.0 (quilt) Date: Wed, 15 Feb 2023 10:20:29 +0900 Message-Id: <20230215012034.403356-7-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230215012034.403356-1-masahiroy@kernel.org> References: <20230215012034.403356-1-masahiroy@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org Change the source format from "1.0" to "3.0 (quilt)" because it works more cleanly. All files except .config and debian/ go into the orig tarball. Add a single patch, debian/patches/config, and delete the ugly extend-diff-ignore patterns. The debian tarball will be compressed into *.debian.tar.xz by default. If you like to use a different compression mode, you can pass the command line option, DPKG_FLAGS=-Zgzip, for example. The orig tarball only supports gzip for now. The combination of gzip and xz is somewhat clumsy, but it is not a practical problem. Signed-off-by: Masahiro Yamada Reviewed-by: Nicolas Schier --- Changes in v6: - Fix error when KCONFIG_CONFIG is set Changes in v4: - New patch scripts/Makefile.package | 2 +- scripts/package/mkdebian | 41 ++++++++++++++++++---------------------- 2 files changed, 19 insertions(+), 24 deletions(-) diff --git a/scripts/Makefile.package b/scripts/Makefile.package index 80a96eb61426..5538deacb136 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -105,7 +105,7 @@ debian-orig: linux.tar.gz debian PHONY += deb-pkg deb-pkg: debian-orig +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ - --build=source,binary --source-option=-sP -nc -us -uc + --build=source,binary -nc -us -uc PHONY += bindeb-pkg bindeb-pkg: debian diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian index 534e9713761a..22fc73a5e4f2 100755 --- a/scripts/package/mkdebian +++ b/scripts/package/mkdebian @@ -135,29 +135,24 @@ else fi mkdir -p debian/source/ -echo "1.0" > debian/source/format - -# Ugly: ignore anything except .config or debian/ -cat<<'EOF' > debian/source/local-options -diff-ignore - -extend-diff-ignore = ^[^.d] - -extend-diff-ignore = ^\.[^c] -extend-diff-ignore = ^\.c($|[^o]) -extend-diff-ignore = ^\.co($|[^n]) -extend-diff-ignore = ^\.con($|[^f]) -extend-diff-ignore = ^\.conf($|[^i]) -extend-diff-ignore = ^\.confi($|[^g]) -extend-diff-ignore = ^\.config. - -extend-diff-ignore = ^d($|[^e]) -extend-diff-ignore = ^de($|[^b]) -extend-diff-ignore = ^deb($|[^i]) -extend-diff-ignore = ^debi($|[^a]) -extend-diff-ignore = ^debia($|[^n]) -extend-diff-ignore = ^debian[^/] -EOF +echo "3.0 (quilt)" > debian/source/format + +{ + echo "diff-ignore" + echo "extend-diff-ignore = .*" +} > debian/source/local-options + +# Add .config as a patch +mkdir -p debian/patches +{ + echo "Subject: Add .config" + echo "Author: ${maintainer}" + echo + echo "--- /dev/null" + echo "+++ linux/.config" + diff -u /dev/null "${KCONFIG_CONFIG}" | tail -n +3 +} > debian/patches/config +echo config > debian/patches/series echo $debarch > debian/arch extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)" From patchwork Wed Feb 15 01:20:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 13141178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC788C61DA4 for ; Wed, 15 Feb 2023 01:23:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233643AbjBOBXP (ORCPT ); Tue, 14 Feb 2023 20:23:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233645AbjBOBXC (ORCPT ); Tue, 14 Feb 2023 20:23:02 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E81931E04; Tue, 14 Feb 2023 17:22:15 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D3769B81F88; Wed, 15 Feb 2023 01:21:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0E33BC4339E; Wed, 15 Feb 2023 01:21:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676424106; bh=xoiM6mWes29Mf1DMrYU7f872NlsvnuFCR71OIJFjM4o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iKhDw9TF0XC+O2zLcgg8lApTf6z1ja4d0VaqGEyhzjIVCVWDLvgcZSHfpLlbfa919 F6fZ1fk75YaXiVm3kLtWz6lmhQ6fCgCbH6fCNRdpZBCSVc2TP2GwITBBF5BycMF31S eUAKt1TI6iMR2UGSj/n9vEhoCdOGqfneO3bhzCSDhCPwvYsXJsKtvdAdcOnrMCvzTh 2r95oiedrUwQFamW/aBYqzAcMxY6qM7AHzkdh3nkecwaGACcYeEiOAp70iJNK77NxY 6/rzm/E/ed9j/6HWGjBckwcBB1ZV34Drkq4i2KoE1R7m4WkC5swJ/2ajHQsD9Ddkcs pAcRuV+S0LtMQ== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ben Hutchings , Masahiro Yamada , Nathan Chancellor , Nick Desaulniers , Nicolas Schier Subject: [PATCH v6 08/12] kbuild: make perf-tar*-src-pkg work without relying on git Date: Wed, 15 Feb 2023 10:20:30 +0900 Message-Id: <20230215012034.403356-8-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230215012034.403356-1-masahiroy@kernel.org> References: <20230215012034.403356-1-masahiroy@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org Currently, perf-tar*-src-pkg only uses 'git archive', but it is better to make it work without relying on git. The file, HEAD, which saves the commit hash, will be included in the tarball only when the source tree is managed by git. The git tree is more precisely checked; it has been copied from scripts/setlocalversion. Signed-off-by: Masahiro Yamada --- (no changes since v5) Changes in v5: - New patch scripts/Makefile.package | 97 ++++++++++++++++++++++++++-------------- 1 file changed, 64 insertions(+), 33 deletions(-) diff --git a/scripts/Makefile.package b/scripts/Makefile.package index 5538deacb136..50b86b325d7a 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -50,10 +50,25 @@ quiet_cmd_tar = TAR $@ tar-rootdir := $(srctree) +%.tar: + $(call cmd,tar) + %.tar.gz: private tar-compress-opt := -I $(KGZIP) %.tar.gz: $(call cmd,tar) +%.tar.bz2: private tar-compress-opt := -I $(KBZIP2) +%.tar.bz2: + $(call cmd,tar) + +%.tar.xz: private tar-compress-opt := -I $(XZ) +%.tar.xz: + $(call cmd,tar) + +%.tar.zst: private tar-compress-opt := -I $(ZSTD) +%.tar.zst: + $(call cmd,tar) + # Linux source tarball # --------------------------------------------------------------------------- @@ -138,36 +153,52 @@ $(tar-pkgs): $(MAKE) -f $(srctree)/Makefile +$(CONFIG_SHELL) $(srctree)/scripts/package/buildtar $@ -# perf-pkg - generate a source tarball with perf source +# perf-tar*-src-pkg - generate a source tarball with perf source # --------------------------------------------------------------------------- -perf-tar=perf-$(KERNELVERSION) - -quiet_cmd_perf_tar = TAR - cmd_perf_tar = \ -git --git-dir=$(srctree)/.git archive --prefix=$(perf-tar)/ \ - HEAD^{tree} $$(cd $(srctree); \ - echo $$(cat tools/perf/MANIFEST)) \ - -o $(perf-tar).tar; \ -mkdir -p $(perf-tar); \ -git --git-dir=$(srctree)/.git rev-parse HEAD > $(perf-tar)/HEAD; \ -(cd $(srctree)/tools/perf; \ -util/PERF-VERSION-GEN $(CURDIR)/$(perf-tar)/); \ -tar rf $(perf-tar).tar $(perf-tar)/HEAD $(perf-tar)/PERF-VERSION-FILE; \ -rm -r $(perf-tar); \ -$(if $(findstring tar-src,$@),, \ -$(if $(findstring bz2,$@),$(KBZIP2), \ -$(if $(findstring gz,$@),$(KGZIP), \ -$(if $(findstring xz,$@),$(XZ), \ -$(if $(findstring zst,$@),$(ZSTD), \ -$(error unknown target $@))))) \ - -f -9 $(perf-tar).tar) - -perf-tar-pkgs := perf-tar-src-pkg perf-targz-src-pkg perf-tarbz2-src-pkg \ - perf-tarxz-src-pkg perf-tarzst-src-pkg -PHONY += $(perf-tar-pkgs) -$(perf-tar-pkgs): - $(call cmd,perf_tar) +perf-tar-src-pkg-tarball = perf-$(KERNELVERSION).$(1) +perf-tar-src-pkg-phony = perf-$(subst .,,$(1))-src-pkg + +quiet_cmd_stage_perf_src = STAGE $@ + cmd_stage_perf_src = \ + rm -rf $@; \ + mkdir -p $@; \ + tar -c -f - --exclude-from=$<_exclude -C $(srctree) --files-from=$(srctree)/tools/perf/MANIFEST | \ + tar -x -f - -C $@ + +.tmp_perf: .tmp_filelist + $(call cmd,stage_perf_src) + +filechk_perf_head = \ + if test -z "$(git -C $(srctree) rev-parse --show-cdup 2>/dev/null)" && \ + head=$$(git -C $(srctree) rev-parse --verify HEAD 2>/dev/null); then \ + echo $$head; \ + else \ + echo "not a git tree"; \ + fi + +.tmp_perf/HEAD: .tmp_perf FORCE + $(call filechk,perf_head) + +quiet_cmd_perf_version_file = GEN $@ + cmd_perf_version_file = cd $(srctree)/tools/perf; util/PERF-VERSION-GEN $(dir $(abspath $@)) + +# PERF-VERSION-FILE and HEAD are independent, but this avoids updating the +# timestamp of PERF-VERSION-FILE. +# The best is to fix tools/perf/util/PERF-VERSION-GEN. +.tmp_perf/PERF-VERSION-FILE: .tmp_perf/HEAD $(srctree)/tools/perf/util/PERF-VERSION-GEN + $(call cmd,perf_version_file) + +define perf-tar-src-pkg-rule +PHONY += $(perf-tar-src-pkg-phony) +$(perf-tar-src-pkg-phony): $(perf-tar-src-pkg-tarball) + @: + +$(perf-tar-src-pkg-tarball): private tar-rootdir := .tmp_perf +$(perf-tar-src-pkg-tarball): .tmp_filelist .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE +endef + +$(foreach x, tar tar.gz tar.bz2 tar.xz tar.zst, $(eval $(call perf-tar-src-pkg-rule,$(x)))) # Help text displayed when executing 'make help' # --------------------------------------------------------------------------- @@ -186,11 +217,11 @@ help: @echo ' tarbz2-pkg - Build the kernel as a bzip2 compressed tarball' @echo ' tarxz-pkg - Build the kernel as a xz compressed tarball' @echo ' tarzst-pkg - Build the kernel as a zstd compressed tarball' - @echo ' perf-tar-src-pkg - Build $(perf-tar).tar source tarball' - @echo ' perf-targz-src-pkg - Build $(perf-tar).tar.gz source tarball' - @echo ' perf-tarbz2-src-pkg - Build $(perf-tar).tar.bz2 source tarball' - @echo ' perf-tarxz-src-pkg - Build $(perf-tar).tar.xz source tarball' - @echo ' perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball' + @echo ' perf-tar-src-pkg - Build the perf source tarball with no compression' + @echo ' perf-targz-src-pkg - Build the perf source tarball with gzip compression' + @echo ' perf-tarbz2-src-pkg - Build the perf source tarball with bz2 compression' + @echo ' perf-tarxz-src-pkg - Build the perf source tarball with xz compression' + @echo ' perf-tarzst-src-pkg - Build the perf source tarball with zst compression' PHONY += FORCE FORCE: From patchwork Wed Feb 15 01:20:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 13141174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06EA2C6379F for ; Wed, 15 Feb 2023 01:22:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233472AbjBOBWg (ORCPT ); Tue, 14 Feb 2023 20:22:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233491AbjBOBWU (ORCPT ); Tue, 14 Feb 2023 20:22:20 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5FBD432CD2; Tue, 14 Feb 2023 17:21:49 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id ED1076199C; Wed, 15 Feb 2023 01:21:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F0DBCC433D2; Wed, 15 Feb 2023 01:21:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676424108; bh=WszWYo1Dp3gtPmdsDTDcLO4U185HID5tYaADLVl+YJ8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tuXS3J3lUbtayvg3n33el4JAI3hK8xT9eZMWTj6IPX1KusGuisadhyvWNeu5MGZh5 wFDikPC6YAbtPrvVF0ls/l2XO93KH2ogMTlvoRm8g3//MatM+WVGiEy1dqnsETc6bE qiHXJrxo5cCvqh/lHPXO+fFmOot3/DrhSXEbUo0i6K2lqjBs15VYDMxobws3pweOuG dqkl2/QnvzWD9h6wgefhNvPpg7Hf3M4hGpsnTfqyIhZIMRYEvsOTvT2FusR7JvHpUp y7cV2B5Xh83kEp3LuFB4DWn9+9+QqdhJq7rLSOBwHcl+TBNzp+DPq2H8amFcQNprLJ 3hmB9vs4eI7PQ== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ben Hutchings , Masahiro Yamada , Nathan Chancellor , Nick Desaulniers , Nicolas Schier Subject: [PATCH v6 09/12] kbuild: tar-pkg: use tar rules in scripts/Makefile.package Date: Wed, 15 Feb 2023 10:20:31 +0900 Message-Id: <20230215012034.403356-9-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230215012034.403356-1-masahiroy@kernel.org> References: <20230215012034.403356-1-masahiroy@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org Use %.tar, %.tar.gz, %.tar.bz2, %.tar.xz, %.tar.zst rules in scripts/Makefile.package. Signed-off-by: Masahiro Yamada --- Changes in v6: - New patch scripts/Makefile.package | 29 +++++++++++++++++----- scripts/package/buildtar | 52 +--------------------------------------- 2 files changed, 24 insertions(+), 57 deletions(-) diff --git a/scripts/Makefile.package b/scripts/Makefile.package index 50b86b325d7a..b941e6341b36 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -145,13 +145,30 @@ snap-pkg: cd $(objtree)/snap && \ snapcraft --target-arch=$(UTS_MACHINE) -# tarball targets +# dir-pkg tar*-pkg - tarball targets # --------------------------------------------------------------------------- -tar-pkgs := dir-pkg tar-pkg targz-pkg tarbz2-pkg tarxz-pkg tarzst-pkg -PHONY += $(tar-pkgs) -$(tar-pkgs): - $(MAKE) -f $(srctree)/Makefile - +$(CONFIG_SHELL) $(srctree)/scripts/package/buildtar $@ + +tar-pkg-tarball = linux-$(KERNELRELEASE)-$(ARCH).$(1) +tar-pkg-phony = $(subst .,,$(1))-pkg + +tar-install: FORCE + $(Q)$(MAKE) -f $(srctree)/Makefile + +$(Q)$(srctree)/scripts/package/buildtar $@ + +PHONY += dir-pkg +dir-pkg: tar-install + @echo "Kernel tree successfully created in $<" + +define tar-pkg-rule +PHONY += $(tar-pkg-phony) +$(tar-pkg-phony): $(tar-pkg-tarball) + @: + +$(tar-pkg-tarball): private tar-rootdir := tar-install +$(tar-pkg-tarball): tar-install +endef + +$(foreach x, tar tar.gz tar.bz2 tar.xz tar.zst, $(eval $(call tar-pkg-rule,$(x)))) # perf-tar*-src-pkg - generate a source tarball with perf source # --------------------------------------------------------------------------- diff --git a/scripts/package/buildtar b/scripts/package/buildtar index 4d6f0b128efd..65b4ea502962 100755 --- a/scripts/package/buildtar +++ b/scripts/package/buildtar @@ -16,39 +16,7 @@ set -e # # Some variables and settings used throughout the script # -tmpdir="${objtree}/tar-install" -tarball="${objtree}/linux-${KERNELRELEASE}-${ARCH}.tar" - - -# -# Figure out how to compress, if requested at all -# -case "${1}" in - dir-pkg|tar-pkg) - opts= - ;; - targz-pkg) - opts="-I ${KGZIP}" - tarball=${tarball}.gz - ;; - tarbz2-pkg) - opts="-I ${KBZIP2}" - tarball=${tarball}.bz2 - ;; - tarxz-pkg) - opts="-I ${XZ}" - tarball=${tarball}.xz - ;; - tarzst-pkg) - opts="-I ${ZSTD}" - tarball=${tarball}.zst - ;; - *) - echo "Unknown tarball target \"${1}\" requested, please add it to ${0}." >&2 - exit 1 - ;; -esac - +tmpdir=$1 # # Clean-up and re-create the temporary directory @@ -148,21 +116,3 @@ case "${ARCH}" in sleep 5 ;; esac - -if [ "${1}" = dir-pkg ]; then - echo "Kernel tree successfully created in $tmpdir" - exit 0 -fi - -# -# Create the tarball -# -if tar --owner=root --group=root --help >/dev/null 2>&1; then - opts="$opts --owner=root --group=root" -fi - -tar cf $tarball -C $tmpdir $opts $dirs - -echo "Tarball successfully created in $tarball" - -exit 0 From patchwork Wed Feb 15 01:20:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 13141175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E403C61DA4 for ; Wed, 15 Feb 2023 01:22:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233493AbjBOBWo (ORCPT ); Tue, 14 Feb 2023 20:22:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233519AbjBOBWU (ORCPT ); Tue, 14 Feb 2023 20:22:20 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E88534329; Tue, 14 Feb 2023 17:21:51 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CEE0F61995; Wed, 15 Feb 2023 01:21:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D17A1C4339B; Wed, 15 Feb 2023 01:21:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676424110; bh=6sFDo6FafZi154G+JCbShxczRgVL1UN+FRTvqIYv8wI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZDqGKl+e1zll1x7Oa+77V7fifFvYMVKuuRpY6xxqqIzJVqVtCivlOHhSKJ4fjYL8c V63fs825mPcRJFVNaUUYLfKKUm38cmk8uFwnZbZxmp2yspSz9nfBGPqJrjgdp5kNGv byQ9pJEpJ27i21kmv+qi8JK3yKHTyl3Lcpb0p+ULApO5Zl+DocQ5cNOSmcu/1GN2b6 p1P4tNdwogGihrezA+m5GgRt1uByym4FcNE+spKDypMkJFOjsxAEko3btGW+IcvJRI opWem93UZ1WKTc5SWkLHnX+aplDKVTd5ESJtlsgmykFeo411uTMycO+YyAImm6iPks yPlF1EaZSK8ZA== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ben Hutchings , Masahiro Yamada , Nathan Chancellor , Nick Desaulniers , Nicolas Schier Subject: [PATCH v6 10/12] kbuild: deb-pkg: fix binary-arch and clean in debian/rules Date: Wed, 15 Feb 2023 10:20:32 +0900 Message-Id: <20230215012034.403356-10-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230215012034.403356-1-masahiroy@kernel.org> References: <20230215012034.403356-1-masahiroy@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org The clean target needs ARCH=${ARCH} to clean up the tree for the correct architecture. 'make (bin)deb-pkg' skips cleaning, but the preclean hook may be executed if dpkg-buildpackage is directly used. The binary-arch target does not need KERNELRELEASE because it is not updated during the installation. KBUILD_BUILD_VERSION is not needed either because binary-arch does not build vmlinux. Signed-off-by: Masahiro Yamada --- Changes in v6: - New patch scripts/package/mkdebian | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian index 22fc73a5e4f2..eef20d5dc288 100755 --- a/scripts/package/mkdebian +++ b/scripts/package/mkdebian @@ -251,12 +251,10 @@ build: build-arch binary-indep: binary-arch: build-arch - \$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \ - KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile intdeb-pkg - + \$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} intdeb-pkg clean: rm -rf debian/files debian/linux-* - \$(MAKE) clean + \$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} clean binary: binary-arch EOF From patchwork Wed Feb 15 01:20:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 13141176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C162DC05027 for ; Wed, 15 Feb 2023 01:22:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233608AbjBOBWu (ORCPT ); Tue, 14 Feb 2023 20:22:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233501AbjBOBWg (ORCPT ); Tue, 14 Feb 2023 20:22:36 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 42B9C34C12; Tue, 14 Feb 2023 17:21:53 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A83B86197C; Wed, 15 Feb 2023 01:21:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AA7EBC433D2; Wed, 15 Feb 2023 01:21:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676424112; bh=FeChVdeQN1bgxPs5mJTWw9fvEMDfbK6uQA/n72DiAws=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Qt+Wbrl84eyJmA99AQr0Rc42SzoZMETg5jj+aNK2ecAcg9UgANmfkOzMx0T2tyv65 ocSizuZyPfTEP/xmMxWDl4W/eloh2O5XS3oa/CfxvSxMN84fBM9z6l6xl+ij6QN5Xi vec+W3pKT/qGSrjvDAUiSrpbiJ3X9xMHrxmiZ/+aizfmYChNr2NNGmOJ0PehTYDBIB 4Kz3jnJiKsGyIt9z6zud54ybNgnKzhUW9ObsLR1Obon13/0gUN+DsA9x+L0/3al2IN 9Tf54kcEwAQBueHHZJ7zsnk+Oe6pg1eFi/rKCd4xxYpIF9rbh8bXh7Dbjd9ZkKMJys ztUvcGsY3T/EQ== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ben Hutchings , Masahiro Yamada , Nathan Chancellor , Nick Desaulniers , Nicolas Schier Subject: [PATCH v6 11/12] kbuild: deb-pkg: improve the usability of source package Date: Wed, 15 Feb 2023 10:20:33 +0900 Message-Id: <20230215012034.403356-11-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230215012034.403356-1-masahiroy@kernel.org> References: <20230215012034.403356-1-masahiroy@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org Improve the source package support in case the dpkg-buildpackage is directly used to build binary packages. For cross-compiling, you can set CROSS_COMPILE via the environment variable, but it is better to set it automatically - set it to ${DEB_HOST_GNU_TYPE}- if we are cross-compiling but not from the top Makefile. The generated source package may be carried to a different build environment, which may have a different compiler installed. Run olddefconfig first to set new CONFIG options to their default values without prompting. Take KERNELRELEASE and KBUILD_BUILD_VERSION from the version field of debian/changelog in case it is updated afterwards. Signed-off-by: Masahiro Yamada --- Changes in v6: - New patch scripts/package/deb-build-option | 16 ++++++++++++++++ scripts/package/mkdebian | 9 ++++----- 2 files changed, 20 insertions(+), 5 deletions(-) create mode 100755 scripts/package/deb-build-option diff --git a/scripts/package/deb-build-option b/scripts/package/deb-build-option new file mode 100755 index 000000000000..b079b0d121d4 --- /dev/null +++ b/scripts/package/deb-build-option @@ -0,0 +1,16 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-only + +# Set up CROSS_COMPILE if we are cross-compiling, but not called from the +# kernel toplevel Makefile +if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUILD_ARCH}" ]; then + echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}- +fi + +version=$(dpkg-parsechangelog -S Version) +version_upstream="${version%-*}" +debian_revision="${version#${version_upstream}}" +debian_revision="${debian_revision#*-}" + +echo KERNELRELEASE=${version_upstream} +echo KBUILD_BUILD_VERSION=${debian_revision} diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian index eef20d5dc288..f74380036bb5 100755 --- a/scripts/package/mkdebian +++ b/scripts/package/mkdebian @@ -90,10 +90,8 @@ rm -rf debian version=$KERNELRELEASE if [ -n "$KDEB_PKGVERSION" ]; then packageversion=$KDEB_PKGVERSION - revision=${packageversion##*-} else - revision=$($srctree/init/build-version) - packageversion=$version-$revision + packageversion=$version-$($srctree/init/build-version) fi sourcename=${KDEB_SOURCENAME:-linux-upstream} @@ -244,8 +242,9 @@ srctree ?= . build-indep: build-arch: - \$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \ - KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile + \$(MAKE) -f \$(srctree)/Makefile ARCH=${ARCH} \ + \$(shell \$(srctree)/scripts/package/deb-build-option) \ + olddefconfig all build: build-arch From patchwork Wed Feb 15 01:20:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 13141177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 443B3C05027 for ; Wed, 15 Feb 2023 01:22:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233621AbjBOBWx (ORCPT ); Tue, 14 Feb 2023 20:22:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233571AbjBOBWk (ORCPT ); Tue, 14 Feb 2023 20:22:40 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D8B1B3345A; Tue, 14 Feb 2023 17:21:54 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6C72E6198D; Wed, 15 Feb 2023 01:21:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8428FC4339B; Wed, 15 Feb 2023 01:21:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676424113; bh=3R0g8mH0HfVeY9FreGkKJ/GZs7e+F2BbvGb/IvPgPy0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=D48JCem6bygp0I6spR8PbF1EHyZyFI1QA8uihjpdnRla/umK2r2AQKWIOL+OUlkyg /Ym5JV1JbC+rVTGMNgWF1cNQSkxio2fXpXC/RabTQ0yjetzAWsuRrTgw9Wkfui3goI AhyEXZB+tW701Oq5pO48CMnkU6wVyStlWWQt6MM7S4e3CVofMVk8fqcqbWWn/SrmJ8 64UV7VFJt6gUnWTgeqGaVe9NObLL98VRQZF06F4kkGCT7SfpB+rCrv1SFCTJHOcdZd erCACdaLNtT56XUADTUlZcMth0LHEv6ecKlZLYtdVDRUyqvsMBjp2/y0nIKyjymxTr G+SwMQa21l3WQ== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ben Hutchings , Masahiro Yamada , Nathan Chancellor , Nick Desaulniers , Nicolas Schier Subject: [PATCH v6 12/12] kbuild: add srcdeb-pkg target Date: Wed, 15 Feb 2023 10:20:34 +0900 Message-Id: <20230215012034.403356-12-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230215012034.403356-1-masahiroy@kernel.org> References: <20230215012034.403356-1-masahiroy@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org This new target builds only the debian source package. Factor similar code out to the 'build-deb-pkg' variable. Signed-off-by: Masahiro Yamada Reviewed-by: Nicolas Schier --- Changes in v6: - New patch scripts/Makefile.package | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/scripts/Makefile.package b/scripts/Makefile.package index b941e6341b36..8650d9432c26 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -100,6 +100,9 @@ binrpm-pkg: +rpmbuild $(RPMOPTS) --define "_builddir $(objtree)" --target \ $(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec +# deb-pkg srcdeb-pkg bindeb-pkg +# --------------------------------------------------------------------------- + quiet_cmd_debianize = GEN $@ cmd_debianize = $(srctree)/scripts/package/mkdebian @@ -117,14 +120,27 @@ debian-orig: linux.tar.gz debian cp $< ../$(orig-name); \ fi +build-deb-pkg = \ + dpkg-buildpackage \ + --build=$(build-type) --no-pre-clean --unsigned-changes \ + $(if $(findstring source, $(build-type)), --unsigned-source) \ + $(if $(findstring binary, $(build-type)), -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch)) \ + $(DPKG_FLAGS) + PHONY += deb-pkg +deb-pkg: private build-type := source,binary deb-pkg: debian-orig - +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) \ - --build=source,binary -nc -us -uc + +$(build-deb-pkg) + +PHONY += srcdeb-pkg +srcdeb-pkg: private build-type := source +srcdeb-pkg: debian-orig + +$(build-deb-pkg) PHONY += bindeb-pkg +bindeb-pkg: private build-type := binary bindeb-pkg: debian - +dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -b -nc -uc + +$(build-deb-pkg) PHONY += intdeb-pkg intdeb-pkg: @@ -225,6 +241,7 @@ help: @echo ' srcrpm-pkg - Build only the source kernel RPM package' @echo ' binrpm-pkg - Build only the binary kernel RPM package' @echo ' deb-pkg - Build both source and binary deb kernel packages' + @echo ' binsrc-pkg - Build only the source kernel deb package' @echo ' bindeb-pkg - Build only the binary kernel deb package' @echo ' snap-pkg - Build only the binary kernel snap package' @echo ' (will connect to external hosts)'