From patchwork Sat Sep 19 21:23:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ren=C3=A9_Scharfe?= X-Patchwork-Id: 11787291 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 51C00139A for ; Sat, 19 Sep 2020 21:23:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1ACA823119 for ; Sat, 19 Sep 2020 21:23:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=web.de header.i=@web.de header.b="s2Swp8pD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726621AbgISVXq (ORCPT ); Sat, 19 Sep 2020 17:23:46 -0400 Received: from mout.web.de ([212.227.17.11]:40065 "EHLO mout.web.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726528AbgISVXp (ORCPT ); Sat, 19 Sep 2020 17:23:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1600550613; bh=1BqCjG+NV2K2dt+nZeUzIxnRFvQJZamXQJ6z8nyhVQs=; h=X-UI-Sender-Class:Subject:From:To:References:Date:In-Reply-To; b=s2Swp8pDDkiPlirUu4yCB37mtESmt1NchGZP/8KorIgVzVh21Tx1t52ftzpNujzDp JHhsAuec7a7ZXJ0dl5Rzgd+Mew9APeXUAYsgUmG/sRTr+m47WyQ4pwBRqOZadOEE/3 3nodkxalqtQbx1a3+R/B3Ud7rfhawKuuAORR2EZc= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from [192.168.178.26] ([91.47.149.245]) by smtp.web.de (mrweb105 [213.165.67.124]) with ESMTPSA (Nemesis) id 1MLRYf-1k1nny0FT1-00IYc3; Sat, 19 Sep 2020 23:23:33 +0200 Subject: [PATCH 1/3] archive: read short blobs in archive.c::write_archive_entry() From: =?utf-8?q?Ren=C3=A9_Scharfe?= To: Junio C Hamano , git@vger.kernel.org References: Message-ID: <0951939b-7521-cf9b-1459-26d6d6774589@web.de> Date: Sat, 19 Sep 2020 23:23:32 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Provags-ID: V03:K1:ryVfkt23TDNYCChnlKqSSAB8CBn0ccvzuCOfo8OG00iJri1Emiq hdHrOY3kejPG3YM6P36QC7l01S2CpG9EM/ta19YEseW24Xzc61qYGz5/i1VbkPSRWEM9TcM IcJHpz1FDhg6AhVvKXhFKdgh/CBePPtImqfg4xYZbkyebBRCt+ifrEbO+CLGIV5L9RSbeW2 dMi3Lm+M4FEO2iur03hDQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:5nJTM07cTyM=:+atrjAFr5zXXo4tfcZ0N9w cqUXsU1p3/2eyERbhr0jDjb+8yTrN/OT5XK+dp/U1u5QUBYv11TPqYiA2Hlu6wsU4NtfX6Rk4 iQdV1/B4qpF2XzvkCw3x2swm2iKYE1x4ceFeUzefMruZD/AfTx7yvkzqyJkK0EVVcJwFnzOBM 0+WgMpXzDWfsgM/1ykz4p68nPFQu0nGqf5V8np+0j7mggZ3voAy75Hc52vBLY/mV92rPVU5zR H+Zr1jpeMrHbVw/culna5j9KnbiWFgJjdAc+pbKB6raG5qdVITqAuQbCNBd1Olv8dRhFBZB2a Fj4cqnu4Hugd5+4fbT9+28V/g3naqMCP05hLpMfFNNCB4JyZOF2CfGJ0Zd4LLi8mPiSCBUuzg SSP4jTPTIBJKZGz/xvtXC9OZKn4Y6gCSScmp2qB8IP06tEB8USEPqLU9jRxeKCCN1NR9MUnqa LIe4S7+TTuA5oR+QHhPM/6kvVIxpWzDpnCFP9bv3/J/9Vvo8bnTwt578d77fJ6gIGXLajAXso tdtzVmcJXZvJQSz0XlmqVQ1/rztArC8x9SK+2ZwED/phi7evIiMAt/w82KSSEzPrPw7FCmd5b YG0eZtag6Z+ildo6zrH9QbsWXMyUglZY2QRcbrgLTuGTUWeU5HerCFHtdXNTw/mWu84tKtDsX 2iqGzaXItGDok2rGisBQMEQnesV8iWWBTDK58yQ7+2KRXbjyKMqcwv4x20mY17FZyiL0odIRL cmWbt4pTsXI+6wSIYkMnqdRc7WpWhKK4oeIT/JZfRYr+LHriw+M1Z40xY8GnaABD6WAeePsH/ gGhlkMvNOoELUz53jAs+z55OUKBNDS5qbRLB5bLzLmya3WKbE+JW41dsjICN0fi/o3k3yIM/m sEUrH18XjTMjl97fNfL7oJ8YxgkDM/GDe7vP/raaJXwzVb+sKjeybzZLeHlhoslfFaO4M33ot swJRfXtlbLLvlfIuvXs6LctNcJ1upryV67PTcZfHYuzER10JpqHePLVAc/EvKC+FW8huPrHmD pMhszyDBf9kbJVzGy3K+DKOt+Rx/kTYI5v3YvMOTd3SGsKGfA8jC6PKr4XyhUBmRZJ3uXBgwl aBRLX8C/Zwr9jdoO+RzRD0jPnCMHCX0fkfo0IqCvr1m4UlLZKxnXiWxa7E0Rc06nALUuZTDUG kZJ3htSsT6dZB4kq95GZyyHO3wRYmt1egWDLnXazM8aA/TTef/X4SOGropkmnSI3P3xZhpXjX COlHy1FlbrtAFJgwh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Centralize reading of symlink destinations and the contents of regular files that are too small to be streamed. This reduces code duplication and allows future patches to add support for adding non-tracked files to archives. The backends are expected to stream blobs if buffer is NULL. object_file_to_archive() is only called from archive.c and thus no longer exported. Signed-off-by: René Scharfe --- archive-tar.c | 22 +++------------------- archive-zip.c | 22 +++++----------------- archive.c | 31 ++++++++++++++++++++++++------- archive.h | 7 ++----- 4 files changed, 34 insertions(+), 48 deletions(-) -- 2.28.0 diff --git a/archive-tar.c b/archive-tar.c index 5ceec3684b..f1a1447ebd 100644 --- a/archive-tar.c +++ b/archive-tar.c @@ -242,13 +242,12 @@ static void write_extended_header(struct archiver_args *args, static int write_tar_entry(struct archiver_args *args, const struct object_id *oid, const char *path, size_t pathlen, - unsigned int mode) + unsigned int mode, + void *buffer, unsigned long size) { struct ustar_header header; struct strbuf ext_header = STRBUF_INIT; - unsigned int old_mode = mode; - unsigned long size, size_in_header; - void *buffer; + unsigned long size_in_header; int err = 0; memset(&header, 0, sizeof(header)); @@ -282,20 +281,6 @@ static int write_tar_entry(struct archiver_args *args, } else memcpy(header.name, path, pathlen); - if (S_ISREG(mode) && !args->convert && - oid_object_info(args->repo, oid, &size) == OBJ_BLOB && - size > big_file_threshold) - buffer = NULL; - else if (S_ISLNK(mode) || S_ISREG(mode)) { - enum object_type type; - buffer = object_file_to_archive(args, path, oid, old_mode, &type, &size); - if (!buffer) - return error(_("cannot read %s"), oid_to_hex(oid)); - } else { - buffer = NULL; - size = 0; - } - if (S_ISLNK(mode)) { if (size > sizeof(header.linkname)) { xsnprintf(header.linkname, sizeof(header.linkname), @@ -326,7 +311,6 @@ static int write_tar_entry(struct archiver_args *args, else err = stream_blocked(args->repo, oid); } - free(buffer); return err; } diff --git a/archive-zip.c b/archive-zip.c index e9f426298b..2961e01c75 100644 --- a/archive-zip.c +++ b/archive-zip.c @@ -285,7 +285,8 @@ static int entry_is_binary(struct index_state *istate, const char *path, static int write_zip_entry(struct archiver_args *args, const struct object_id *oid, const char *path, size_t pathlen, - unsigned int mode) + unsigned int mode, + void *buffer, unsigned long size) { struct zip_local_header header; uintmax_t offset = zip_offset; @@ -299,10 +300,8 @@ static int write_zip_entry(struct archiver_args *args, enum zip_method method; unsigned char *out; void *deflated = NULL; - void *buffer; struct git_istream *stream = NULL; unsigned long flags = 0; - unsigned long size; int is_binary = -1; const char *path_without_prefix = path + args->baselen; unsigned int creator_version = 0; @@ -328,13 +327,8 @@ static int write_zip_entry(struct archiver_args *args, method = ZIP_METHOD_STORE; attr2 = 16; out = NULL; - size = 0; compressed_size = 0; - buffer = NULL; } else if (S_ISREG(mode) || S_ISLNK(mode)) { - enum object_type type = oid_object_info(args->repo, oid, - &size); - method = ZIP_METHOD_STORE; attr2 = S_ISLNK(mode) ? ((mode | 0777) << 16) : (mode & 0111) ? ((mode) << 16) : 0; @@ -343,21 +337,16 @@ static int write_zip_entry(struct archiver_args *args, if (S_ISREG(mode) && args->compression_level != 0 && size > 0) method = ZIP_METHOD_DEFLATE; - if (S_ISREG(mode) && type == OBJ_BLOB && !args->convert && - size > big_file_threshold) { + if (!buffer) { + enum object_type type; stream = open_istream(args->repo, oid, &type, &size, NULL); if (!stream) return error(_("cannot stream blob %s"), oid_to_hex(oid)); flags |= ZIP_STREAM; - out = buffer = NULL; + out = NULL; } else { - buffer = object_file_to_archive(args, path, oid, mode, - &type, &size); - if (!buffer) - return error(_("cannot read %s"), - oid_to_hex(oid)); crc = crc32(crc, buffer, size); is_binary = entry_is_binary(args->repo->index, path_without_prefix, @@ -511,7 +500,6 @@ static int write_zip_entry(struct archiver_args *args, } free(deflated); - free(buffer); if (compressed_size > 0xffffffff || size > 0xffffffff || offset > 0xffffffff) { diff --git a/archive.c b/archive.c index 0de6048bfc..4fbe5329c5 100644 --- a/archive.c +++ b/archive.c @@ -70,10 +70,12 @@ static void format_subst(const struct commit *commit, free(to_free); } -void *object_file_to_archive(const struct archiver_args *args, - const char *path, const struct object_id *oid, - unsigned int mode, enum object_type *type, - unsigned long *sizep) +static void *object_file_to_archive(const struct archiver_args *args, + const char *path, + const struct object_id *oid, + unsigned int mode, + enum object_type *type, + unsigned long *sizep) { void *buffer; const struct commit *commit = args->convert ? args->commit : NULL; @@ -145,6 +147,9 @@ static int write_archive_entry(const struct object_id *oid, const char *base, write_archive_entry_fn_t write_entry = c->write_entry; int err; const char *path_without_prefix; + unsigned long size; + void *buffer; + enum object_type type; args->convert = 0; strbuf_reset(&path); @@ -167,7 +172,7 @@ static int write_archive_entry(const struct object_id *oid, const char *base, if (S_ISDIR(mode) || S_ISGITLINK(mode)) { if (args->verbose) fprintf(stderr, "%.*s\n", (int)path.len, path.buf); - err = write_entry(args, oid, path.buf, path.len, mode); + err = write_entry(args, oid, path.buf, path.len, mode, NULL, 0); if (err) return err; return (S_ISDIR(mode) ? READ_TREE_RECURSIVE : 0); @@ -175,7 +180,19 @@ static int write_archive_entry(const struct object_id *oid, const char *base, if (args->verbose) fprintf(stderr, "%.*s\n", (int)path.len, path.buf); - return write_entry(args, oid, path.buf, path.len, mode); + + /* Stream it? */ + if (S_ISREG(mode) && !args->convert && + oid_object_info(args->repo, oid, &size) == OBJ_BLOB && + size > big_file_threshold) + return write_entry(args, oid, path.buf, path.len, mode, NULL, size); + + buffer = object_file_to_archive(args, path.buf, oid, mode, &type, &size); + if (!buffer) + return error(_("cannot read %s"), oid_to_hex(oid)); + err = write_entry(args, oid, path.buf, path.len, mode, buffer, size); + free(buffer); + return err; } static void queue_directory(const unsigned char *sha1, @@ -265,7 +282,7 @@ int write_archive_entries(struct archiver_args *args, if (args->verbose) fprintf(stderr, "%.*s\n", (int)len, args->base); err = write_entry(args, &args->tree->object.oid, args->base, - len, 040777); + len, 040777, NULL, 0); if (err) return err; } diff --git a/archive.h b/archive.h index 3bd96bf6bb..d83b41a01f 100644 --- a/archive.h +++ b/archive.h @@ -49,12 +49,9 @@ void init_archivers(void); typedef int (*write_archive_entry_fn_t)(struct archiver_args *args, const struct object_id *oid, const char *path, size_t pathlen, - unsigned int mode); + unsigned int mode, + void *buffer, unsigned long size); int write_archive_entries(struct archiver_args *args, write_archive_entry_fn_t write_entry); -void *object_file_to_archive(const struct archiver_args *args, - const char *path, const struct object_id *oid, - unsigned int mode, enum object_type *type, - unsigned long *sizep); #endif /* ARCHIVE_H */ From patchwork Sat Sep 19 21:23:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ren=C3=A9_Scharfe?= X-Patchwork-Id: 11787293 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 25087746 for ; Sat, 19 Sep 2020 21:23:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0445723119 for ; Sat, 19 Sep 2020 21:23:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=web.de header.i=@web.de header.b="TTsCR9dR" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726660AbgISVXt (ORCPT ); Sat, 19 Sep 2020 17:23:49 -0400 Received: from mout.web.de ([212.227.17.11]:41065 "EHLO mout.web.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726528AbgISVXs (ORCPT ); Sat, 19 Sep 2020 17:23:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1600550623; bh=rfr+VkvdZ9atojVNNx7DBcKKgrzoxtRmnIGIQm0/d1w=; h=X-UI-Sender-Class:Subject:From:To:References:Date:In-Reply-To; b=TTsCR9dRLxs0qa/uUmbd+QMWVaFp64b7C3GrW+5dvZSVsUH7Bub6nCPspqgITpWzB 9oVtTUVCArFr1omactV04x/+eLr+SYAyjisc6aoRTlbLtxtL9X+y43kbNkrtQFI/R4 XnDhpTFjyox51Yg2lC14Rak8leZ2WB6SLU9CKLo8= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from [192.168.178.26] ([91.47.149.245]) by smtp.web.de (mrweb102 [213.165.67.124]) with ESMTPSA (Nemesis) id 0LdF4n-1kkWVi26ZV-00iWoV; Sat, 19 Sep 2020 23:23:43 +0200 Subject: [PATCH 2/3] archive: add --add-file From: =?utf-8?q?Ren=C3=A9_Scharfe?= To: Junio C Hamano , git@vger.kernel.org References: Message-ID: <1c03ea9b-ec91-9a07-781f-626530ac5a8d@web.de> Date: Sat, 19 Sep 2020 23:23:42 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Provags-ID: V03:K1:lZgNEVe79xOYG+MbeIk7jBEW7ebJYEcJtLjZ4nP79Gx3jzMK4Md SM4uN64cVBYnDoq2+o6Nxb+W6S90RSnMbzL5HxTmMXOO7Eg4XJha1BIa5pKNRDWu3AUlMyN YXTRYAdrxxz0OQf5urv5gr0dkQX8deNAU0cEgn1v1uK183K+bsHU1o4hEnFuNOMWvATA7ye HLTDIUZZp9uaiq86fa4wg== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:KqXirrGr9TE=:jkFphmRQuST09W/CCTVjkM F4K6GQfyanj97yrJI+KuxKzq+TDjBCnLfVkUqH/MhiC2r2/Rv0hz1vuF4TFtXgwAiHdpzfxnJ vivykvQ+LkRm+wmm6rFrZAbSNm0wEx/6dWlOLbsEdNrHCeMrYY9KkDmrgiTs60HRtJDVkQLpu dw6z/ox4xdTDCDXeCUldbXeIY0waNLFXzJ4H8fdaeqoyUJKUNlgT7AetX6NUgVgYqDkpZBjdi a0D4mJ5ZXtU1ZP8j/g6gDk6Y3PixQ4CZLkLC2FFOhKSVfw3Y3EK9u1TCIZR371HYe+SY0hO0b vdlsUJw1uQkgQ5PovcjPdP6chfbvBFDrUBPTH4OTynvDd0OL1fjjB2LgUIXJh5yoTSQMAZBw7 kdR6AEcw3RMf3kxFtTWHJl/nrpxcpEmpc/bDDepbbI8dgixLhqBvG9P4iTmwPRF7j+PYqrwB9 ZSFdv4ba/ICwfm8mUjhXttralMdVBpzqCAKhUtu6TA98RVB9+VZnkycPFspOWprtlMtYPpKQu bjXB/yB4rw5907X1uuIXNWrJ8PtZ8ouRD0jV7NBQxNSNdT4mKmCBn0iIVuZxlpyE2B8GtPiL9 3r1Ex91baQGr7i3A5sLjbu/1rgG9mDCtc2Vku7EFTa3W1V+3k8jRFM9ZoDXUROquPtQ1k0YZ9 IuuYbltk/rRLeAtmsHB3ZBmJbw4ziSPgCA9O8rayc4urNd1n37lFP0+kPdmVHSvNEmOFxLYFs ZjzyAnDNM/2gfbGcJ8Dex1fUagWshOCP652sVuCzoFCpAQKrHPDLUnfb2k0i1etIlgB5UusWr TMXXVIsoScr6xTHsfpnMxe7YOc3Z8tu5xE3W/sFUf8c4Qz9AjEYy5QxagLVI95RzZRKO3bM8v s94Lctm9E04NxagDHFtBkJ1GqQAvu7IlAC1dmlgr+Bih6l4dizfAKS8CdHhU+jJHVt1TvveqL VOQSSOsB8Y56C5d6JGcjxNo8Q9LIewl2MbvebabobgUfNcCORXh6wOTv/u2DZijKKb/9Kc9HP CYY7B21NmgcLDWYAgB7mefSgoeuKXZ0p6AhiOYiTfwqP4ubaRSeaxPE8xB3ZNmAftIJJJ9bqN ViNSFk52iyHo9R9gryQpV88MBn0n9uiJN3AeeINnvKcTAAcM6f4wQN5LgJlZrIbFKJavnP4St HCKhNtjRO5UVNsaF0s+6oGP3Q4gJF9WkJXzkiIjl/cIc4wDki4h/oMDJa9PdrfbRaaDyIYPIk hy+/Pik2/AWVa8+Ol Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Allow users to append non-tracked files. This simplifies the generation of source packages with a few extra files, e.g. containing version information. They get the same access times and user information as tracked files. Signed-off-by: René Scharfe --- Documentation/git-archive.txt | 6 +++ archive.c | 86 ++++++++++++++++++++++++++++++++++- archive.h | 2 + t/t5000-tar-tree.sh | 29 ++++++++++++ t/t5003-archive-zip.sh | 28 ++++++++++++ 5 files changed, 150 insertions(+), 1 deletion(-) -- 2.28.0 diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt index cfa1e4ebe4..9f8172828d 100644 --- a/Documentation/git-archive.txt +++ b/Documentation/git-archive.txt @@ -55,6 +55,12 @@ OPTIONS --output=:: Write the archive to instead of stdout. +--add-file=:: + Add a non-tracked file to the archive. Can be repeated to add + multiple files. The path of the file in the archive is built + by concatenating the value for `--prefix` (if any) and the + basename of . + --worktree-attributes:: Look for attributes in .gitattributes files in the working tree as well (see <>). diff --git a/archive.c b/archive.c index 4fbe5329c5..3c1541af9e 100644 --- a/archive.c +++ b/archive.c @@ -266,6 +266,11 @@ static int queue_or_write_archive_entry(const struct object_id *oid, stage, context); } +struct extra_file_info { + char *base; + struct stat stat; +}; + int write_archive_entries(struct archiver_args *args, write_archive_entry_fn_t write_entry) { @@ -273,6 +278,10 @@ int write_archive_entries(struct archiver_args *args, struct unpack_trees_options opts; struct tree_desc t; int err; + struct strbuf path_in_archive = STRBUF_INIT; + struct strbuf content = STRBUF_INIT; + struct object_id fake_oid = null_oid; + int i; if (args->baselen > 0 && args->base[args->baselen - 1] == '/') { size_t len = args->baselen; @@ -318,6 +327,33 @@ int write_archive_entries(struct archiver_args *args, free(context.bottom); context.bottom = next; } + + for (i = 0; i < args->extra_files.nr; i++) { + struct string_list_item *item = args->extra_files.items + i; + char *path = item->string; + struct extra_file_info *info = item->util; + + put_be64(fake_oid.hash, i + 1); + + strbuf_reset(&path_in_archive); + if (info->base) + strbuf_addstr(&path_in_archive, info->base); + strbuf_addstr(&path_in_archive, basename(path)); + + strbuf_reset(&content); + if (strbuf_read_file(&content, path, info->stat.st_size) < 0) + err = error_errno(_("could not read '%s'"), path); + else + err = write_entry(args, &fake_oid, path_in_archive.buf, + path_in_archive.len, + info->stat.st_mode, + content.buf, content.len); + if (err) + break; + } + strbuf_release(&path_in_archive); + strbuf_release(&content); + return err; } @@ -457,6 +493,42 @@ static void parse_treeish_arg(const char **argv, ar_args->time = archive_time; } +static void extra_file_info_clear(void *util, const char *str) +{ + struct extra_file_info *info = util; + free(info->base); + free(info); +} + +static int add_file_cb(const struct option *opt, const char *arg, int unset) +{ + struct archiver_args *args = opt->value; + const char **basep = (const char **)opt->defval; + const char *base = *basep; + char *path; + struct string_list_item *item; + struct extra_file_info *info; + + if (unset) { + string_list_clear_func(&args->extra_files, + extra_file_info_clear); + return 0; + } + + if (!arg) + return -1; + + path = prefix_filename(args->prefix, arg); + item = string_list_append_nodup(&args->extra_files, path); + item->util = info = xmalloc(sizeof(*info)); + info->base = xstrdup_or_null(base); + if (stat(path, &info->stat)) + die(_("File not found: %s"), path); + if (!S_ISREG(info->stat.st_mode)) + die(_("Not a regular file: %s"), path); + return 0; +} + #define OPT__COMPR(s, v, h, p) \ OPT_SET_INT_F(s, NULL, v, h, p, PARSE_OPT_NONEG) #define OPT__COMPR_HIDDEN(s, v, p) \ @@ -481,6 +553,9 @@ static int parse_archive_args(int argc, const char **argv, OPT_STRING(0, "format", &format, N_("fmt"), N_("archive format")), OPT_STRING(0, "prefix", &base, N_("prefix"), N_("prepend prefix to each pathname in the archive")), + { OPTION_CALLBACK, 0, "add-file", args, N_("file"), + N_("add untracked file to archive"), 0, add_file_cb, + (intptr_t)&base }, OPT_STRING('o', "output", &output, N_("file"), N_("write the archive to this file")), OPT_BOOL(0, "worktree-attributes", &worktree_attributes, @@ -515,6 +590,8 @@ static int parse_archive_args(int argc, const char **argv, die(_("Option --exec can only be used together with --remote")); if (output) die(_("Unexpected option --output")); + if (is_remote && args->extra_files.nr) + die(_("Options --add-file and --remote cannot be used together")); if (!base) base = ""; @@ -561,11 +638,14 @@ int write_archive(int argc, const char **argv, const char *prefix, { const struct archiver *ar = NULL; struct archiver_args args; + int rc; git_config_get_bool("uploadarchive.allowunreachable", &remote_allow_unreachable); git_config(git_default_config, NULL); args.repo = repo; + args.prefix = prefix; + string_list_init(&args.extra_files, 1); argc = parse_archive_args(argc, argv, &ar, &args, name_hint, remote); if (!startup_info->have_repository) { /* @@ -579,7 +659,11 @@ int write_archive(int argc, const char **argv, const char *prefix, parse_treeish_arg(argv, &args, prefix, remote); parse_pathspec_arg(argv + 1, &args); - return ar->write_archive(ar, &args); + rc = ar->write_archive(ar, &args); + + string_list_clear_func(&args.extra_files, extra_file_info_clear); + + return rc; } static int match_extension(const char *filename, const char *ext) diff --git a/archive.h b/archive.h index d83b41a01f..82b226011a 100644 --- a/archive.h +++ b/archive.h @@ -9,6 +9,7 @@ struct repository; struct archiver_args { struct repository *repo; const char *refname; + const char *prefix; const char *base; size_t baselen; struct tree *tree; @@ -20,6 +21,7 @@ struct archiver_args { unsigned int worktree_attributes : 1; unsigned int convert : 1; int compression_level; + struct string_list extra_files; }; /* main api */ diff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh index 37655a237c..3ebb0d3b65 100755 --- a/t/t5000-tar-tree.sh +++ b/t/t5000-tar-tree.sh @@ -94,6 +94,16 @@ check_tar() { ' } +check_added() { + dir=$1 + path_in_fs=$2 + path_in_archive=$3 + + test_expect_success " validate extra file $path_in_archive" ' + diff -r $path_in_fs $dir/$path_in_archive + ' +} + test_expect_success 'setup' ' test_oid_cache <<-EOF obj sha1:19f9c8273ec45a8938e6999cb59b3ff66739902a @@ -164,6 +174,25 @@ test_expect_success 'git-archive --prefix=olde-' ' check_tar with_olde-prefix olde- +test_expect_success 'git archive --add-file' ' + echo untracked >untracked && + git archive --add-file=untracked HEAD >with_untracked.tar +' + +check_tar with_untracked +check_added with_untracked untracked untracked + +test_expect_success 'git archive --add-file twice' ' + echo untracked >untracked && + git archive --prefix=one/ --add-file=untracked \ + --prefix=two/ --add-file=untracked \ + --prefix= HEAD >with_untracked2.tar +' + +check_tar with_untracked2 +check_added with_untracked2 untracked one/untracked +check_added with_untracked2 untracked two/untracked + test_expect_success 'git archive on large files' ' test_config core.bigfilethreshold 1 && git archive HEAD >b3.tar && diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh index 3b76d2eb65..1e6d18b140 100755 --- a/t/t5003-archive-zip.sh +++ b/t/t5003-archive-zip.sh @@ -72,6 +72,16 @@ check_zip() { " } +check_added() { + dir=$1 + path_in_fs=$2 + path_in_archive=$3 + + test_expect_success UNZIP " validate extra file $path_in_archive" ' + diff -r $path_in_fs $dir/$path_in_archive + ' +} + test_expect_success \ 'populate workdir' \ 'mkdir a && @@ -188,4 +198,22 @@ test_expect_success 'git archive --format=zip on large files' ' check_zip large-compressed +test_expect_success 'git archive --format=zip --add-file' ' + echo untracked >untracked && + git archive --format=zip --add-file=untracked HEAD >with_untracked.zip +' + +check_zip with_untracked +check_added with_untracked untracked untracked + +test_expect_success 'git archive --format=zip --add-file twice' ' + echo untracked >untracked && + git archive --format=zip --prefix=one/ --add-file=untracked \ + --prefix=two/ --add-file=untracked \ + --prefix= HEAD >with_untracked2.zip +' +check_zip with_untracked2 +check_added with_untracked2 untracked one/untracked +check_added with_untracked2 untracked two/untracked + test_done From patchwork Sat Sep 19 21:23:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ren=C3=A9_Scharfe?= X-Patchwork-Id: 11787295 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 76039139A for ; Sat, 19 Sep 2020 21:23:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 579F223119 for ; Sat, 19 Sep 2020 21:23:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=web.de header.i=@web.de header.b="Z3KnUK0n" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726696AbgISVXz (ORCPT ); Sat, 19 Sep 2020 17:23:55 -0400 Received: from mout.web.de ([212.227.17.12]:50845 "EHLO mout.web.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726528AbgISVXz (ORCPT ); Sat, 19 Sep 2020 17:23:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1600550631; bh=IBWQRwIJFEfZ+g3DiwQ6ltOq+qk856XMoRXDL6s2HUg=; h=X-UI-Sender-Class:Subject:From:To:References:Date:In-Reply-To; b=Z3KnUK0nWywdKxfcPMHIvHur92zIRqN91xfjleV5WGPyJsy2u80I6QtELD5stxSe8 6luSBvbIWiOIflbvfMoRGwwrp3fJHm0zy4ntr32dUpFRr/a5YiOLL1jgnJO6yM/Q1Y DPKeufB01ya0EJNx6MUxDEXBJK21nRsHLVszI5sQ= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from [192.168.178.26] ([91.47.149.245]) by smtp.web.de (mrweb105 [213.165.67.124]) with ESMTPSA (Nemesis) id 1N3ouw-1kSZyJ0KVE-00zpmv; Sat, 19 Sep 2020 23:23:51 +0200 Subject: [PATCH 3/3] Makefile: use git-archive --add-file From: =?utf-8?q?Ren=C3=A9_Scharfe?= To: Junio C Hamano , git@vger.kernel.org References: Message-ID: Date: Sat, 19 Sep 2020 23:23:50 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Provags-ID: V03:K1:R7PSy/OolzoQ5xJXbUmhhCXjU5BqARoLzVQBTaj/QQAikY80in7 qsQBQGvIIDzq3P40nYzqp1bfAqNh1KaemcUvkshFbzBotbHEFo8QLyu2gPwf218kccCnuoI 4xgWNUP/k2wkjmYzfKAzSMGy/wn+8URlYncb3oo8zFTFqgXtKWPFpr/RqUC/T4x7cxtY8o8 uvyQsoNimREFK5nHJos3A== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:Fj6//WxjnVI=:4b5ITIOjQ5ohmIpzISYgfq R7KXOeDwwTTPUixgdde5niPYbn4iaMq44tQSlw4+UQaHQnldReREng+cQd/ZCNmwbSYbdk59v sPFicuDHm2iZVDUXhKm88ZJRMSeU/DQJcFbLCfZQp6FAJUhQlbRrOKRrXVGpuNMjxk6y2ZOyB UrsrSpWmMMSt1APyIXDGgVkj2K7EC2khF6liOxj0rMs8vyIj7agv5mvp+VsxwTBrMIxz6+b2O zunuo1X9HFIrsvLlLxB/YpNwETE+n6J+GtCnamRxOs6Sz1YvsT5D7t2ZUAL9CFwLAVTxtJ/Mi q6kfG/ve5pnzUg7HtRhpWELcyh7BLmw1RjLXAlqeEi4EuvevcEPMgcHbShcseCiIDEkJenYMh llbs63P3h7QFVLzwuK8H8YwccEpXlZCTpDPTIECTERwm+zTFu9Xnkk03LEVeibap5oQwzVyFI Dc0C5W7neopjU6h3o4Ikav0VLuM8o6ZcgkSVY4Bot11mR27ilA5m2Eb/A70g7lSS6rxdLgkp9 blJAO+G2mzWRW1wE7znnaFBaap0SHvLCX4h0t+Uv2QrwX72/qHz7Ve8v93A2B1ux8P6XFRNAY 5cwdCU++s2pEB+wfNGuGYtTETNQiN/O3X5ixLqYuTQdF/ZCBe2bBv+YLS59fQ5ENtWo/HQFEL jlI65XhOh0FI5oer2iYk+uUDiyh0+Y5Iudhv9TxgQghLmq54YXp9UyOjt/V1lG+JpTL6NbzHk qFTk9t85ZeqMyu6aYtr08G//aJn2X3MDBEjdn8v2KMIQQoWg+reJ32G+ohGzl3f9bM0UTxrET PwIQI+5+0gyI5TKxnxFpGfC378jV5X3virlVQ5OY8cJ7NRN/8KDdm/8nmIXsaPJAQl3pjIw8o uYhqdZnipbsClcJI0JRNjyWu25AEmohH5gwP0AbqLsSlE9qd12Uzk+OuImaHraIf2CFydDmgS crpM43YVWxfqR+FalFM23mw1hoewGlhmeWSSg9jSbxm9RoKMqtYUQ9IkNXxoe7pFcdS2WlUlR GrJdfQ5OJWztRarpLWxR0JvciSDsbfqLIfz595NP5aSe7rP5/Os8P00yxR/0s1pmxg5I+SMsi kpxYAwpyQqnZf5OBHwh8S80uW1+rimnplPngSn697z9zFLIZUH/E68sNhdWzf1SwG4DKC8+0h eJVwIjtTaI3/bd077/GRJBvbtaL6NFenD2dP6Q/1+mW0e/fCDdtveZNJsSkQIdrD/HcLl1QFl 5ZBdJAl8WbsWHjm97 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add untracked files for the dist target directly using git archive instead of calling tar cr to append them. This reduces the dependency on external tools and gives the untracked files the same access times and user information as tracked ones, integrating them seamlessly. Signed-off-by: René Scharfe --- Makefile | 41 +++++++++++++++++++---------------------- 1 file changed, 19 insertions(+), 22 deletions(-) -- 2.28.0 diff --git a/Makefile b/Makefile index f1b1bc8aa0..56bf12b238 100644 --- a/Makefile +++ b/Makefile @@ -3045,32 +3045,29 @@ quick-install-html: # With GNU tar, "--mode=u+rwX,og+rX,og-w" would be a good idea, for example. TAR_DIST_EXTRA_OPTS = GIT_TARNAME = git-$(GIT_VERSION) +GIT_ARCHIVE_EXTRA_FILES = \ + --prefix=$(GIT_TARNAME)/ \ + --add-file=configure \ + --add-file=$(GIT_TARNAME)/version \ + --prefix=$(GIT_TARNAME)/git-gui/ \ + --add-file=$(GIT_TARNAME)/git-gui/version +ifdef DC_SHA1_SUBMODULE +GIT_ARCHIVE_EXTRA_FILES += \ + --prefix=$(GIT_TARNAME)/sha1collisiondetection/ \ + --add-file=sha1collisiondetection/LICENSE.txt \ + --prefix=$(GIT_TARNAME)/sha1collisiondetection/lib/ \ + --add-file=sha1collisiondetection/lib/sha1.c \ + --add-file=sha1collisiondetection/lib/sha1.h \ + --add-file=sha1collisiondetection/lib/ubc_check.c \ + --add-file=sha1collisiondetection/lib/ubc_check.h +endif dist: git-archive$(X) configure - ./git-archive --format=tar \ - --prefix=$(GIT_TARNAME)/ HEAD^{tree} > $(GIT_TARNAME).tar @mkdir -p $(GIT_TARNAME) - @cp configure $(GIT_TARNAME) @echo $(GIT_VERSION) > $(GIT_TARNAME)/version @$(MAKE) -C git-gui TARDIR=../$(GIT_TARNAME)/git-gui dist-version - $(TAR) rf $(GIT_TARNAME).tar $(TAR_DIST_EXTRA_OPTS) \ - $(GIT_TARNAME)/configure \ - $(GIT_TARNAME)/version \ - $(GIT_TARNAME)/git-gui/version -ifdef DC_SHA1_SUBMODULE - @mkdir -p $(GIT_TARNAME)/sha1collisiondetection/lib - @cp sha1collisiondetection/LICENSE.txt \ - $(GIT_TARNAME)/sha1collisiondetection/ - @cp sha1collisiondetection/LICENSE.txt \ - $(GIT_TARNAME)/sha1collisiondetection/ - @cp sha1collisiondetection/lib/sha1.[ch] \ - $(GIT_TARNAME)/sha1collisiondetection/lib/ - @cp sha1collisiondetection/lib/ubc_check.[ch] \ - $(GIT_TARNAME)/sha1collisiondetection/lib/ - $(TAR) rf $(GIT_TARNAME).tar $(TAR_DIST_EXTRA_OPTS) \ - $(GIT_TARNAME)/sha1collisiondetection/LICENSE.txt \ - $(GIT_TARNAME)/sha1collisiondetection/lib/sha1.[ch] \ - $(GIT_TARNAME)/sha1collisiondetection/lib/ubc_check.[ch] -endif + ./git-archive --format=tar \ + $(GIT_ARCHIVE_EXTRA_FILES) \ + --prefix=$(GIT_TARNAME)/ HEAD^{tree} > $(GIT_TARNAME).tar @$(RM) -r $(GIT_TARNAME) gzip -f -9 $(GIT_TARNAME).tar