From patchwork Thu Apr 4 23:22:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 10886631 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 71BBA139A for ; Thu, 4 Apr 2019 23:22:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5D96828AF2 for ; Thu, 4 Apr 2019 23:22:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5197D28B00; Thu, 4 Apr 2019 23:22:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A214828B17 for ; Thu, 4 Apr 2019 23:22:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729456AbfDDXWF (ORCPT ); Thu, 4 Apr 2019 19:22:05 -0400 Received: from cloud.peff.net ([104.130.231.41]:47392 "HELO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1728721AbfDDXWF (ORCPT ); Thu, 4 Apr 2019 19:22:05 -0400 Received: (qmail 1397 invoked by uid 109); 4 Apr 2019 23:22:05 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with SMTP; Thu, 04 Apr 2019 23:22:05 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 20360 invoked by uid 111); 4 Apr 2019 23:22:32 -0000 Received: from sigill.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.7) by peff.net (qpsmtpd/0.94) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) SMTP; Thu, 04 Apr 2019 19:22:32 -0400 Authentication-Results: peff.net; auth=none Received: by sigill.intra.peff.net (sSMTP sendmail emulation); Thu, 04 Apr 2019 19:22:03 -0400 Date: Thu, 4 Apr 2019 19:22:03 -0400 From: Jeff King To: git@vger.kernel.org Subject: [PATCH 01/12] t5319: fix bogus cat-file argument Message-ID: <20190404232203.GA21839@sigill.intra.peff.net> References: <20190404232104.GA27770@sigill.intra.peff.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190404232104.GA27770@sigill.intra.peff.net> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP There's no such argument as "--unordered"; it's spelled "--unsorted". But our test failed to notice that cat-file didn't run at all because: 1. It lost the exit code of git on the left-hand side of a pipe. 2. It was comparing two runs of the broken invocation with and without a particular config variable (and indeed, both cases produced no output!). Let's fix the option, but also tweak the helper function to check the exit code. Signed-off-by: Jeff King --- t/t5319-multi-pack-index.sh | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 70926b5bc0..42f4d6cd01 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -86,13 +86,14 @@ test_expect_success 'write midx with one v1 pack' ' ' midx_git_two_modes () { + git -c core.multiPackIndex=false $1 >expect && + git -c core.multiPackIndex=true $1 >actual && if [ "$2" = "sorted" ] then - git -c core.multiPackIndex=false $1 | sort >expect && - git -c core.multiPackIndex=true $1 | sort >actual - else - git -c core.multiPackIndex=false $1 >expect && - git -c core.multiPackIndex=true $1 >actual + sort expect.sorted && + mv expect.sorted expect && + sort actual.sorted && + mv actual.sorted actual fi && test_cmp expect actual } @@ -104,7 +105,7 @@ compare_results_with_midx () { midx_git_two_modes "log --raw" && midx_git_two_modes "count-objects --verbose" && midx_git_two_modes "cat-file --batch-all-objects --buffer --batch-check" && - midx_git_two_modes "cat-file --batch-all-objects --buffer --batch-check --unsorted" sorted + midx_git_two_modes "cat-file --batch-all-objects --buffer --batch-check --unordered" sorted ' } From patchwork Thu Apr 4 23:22:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 10886633 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7435213B5 for ; Thu, 4 Apr 2019 23:22:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6164F28ABB for ; Thu, 4 Apr 2019 23:22:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 55AEB28B1F; Thu, 4 Apr 2019 23:22:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7078728ABB for ; Thu, 4 Apr 2019 23:22:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729644AbfDDXWj (ORCPT ); Thu, 4 Apr 2019 19:22:39 -0400 Received: from cloud.peff.net ([104.130.231.41]:47398 "HELO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1729501AbfDDXWj (ORCPT ); Thu, 4 Apr 2019 19:22:39 -0400 Received: (qmail 1408 invoked by uid 109); 4 Apr 2019 23:22:39 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with SMTP; Thu, 04 Apr 2019 23:22:39 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 20376 invoked by uid 111); 4 Apr 2019 23:23:05 -0000 Received: from sigill.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.7) by peff.net (qpsmtpd/0.94) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) SMTP; Thu, 04 Apr 2019 19:23:05 -0400 Authentication-Results: peff.net; auth=none Received: by sigill.intra.peff.net (sSMTP sendmail emulation); Thu, 04 Apr 2019 19:22:37 -0400 Date: Thu, 4 Apr 2019 19:22:37 -0400 From: Jeff King To: git@vger.kernel.org Subject: [PATCH 02/12] t5319: drop useless --buffer from cat-file Message-ID: <20190404232236.GB21839@sigill.intra.peff.net> References: <20190404232104.GA27770@sigill.intra.peff.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190404232104.GA27770@sigill.intra.peff.net> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The cat-file --buffer option is the default already when using --batch-all-objects. It doesn't hurt to specify it, but it's nice for the test scripts to model good usage. Signed-off-by: Jeff King --- t/t5319-multi-pack-index.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 42f4d6cd01..8c4d2bd849 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -104,8 +104,8 @@ compare_results_with_midx () { midx_git_two_modes "rev-list --objects --all" && midx_git_two_modes "log --raw" && midx_git_two_modes "count-objects --verbose" && - midx_git_two_modes "cat-file --batch-all-objects --buffer --batch-check" && - midx_git_two_modes "cat-file --batch-all-objects --buffer --batch-check --unordered" sorted + midx_git_two_modes "cat-file --batch-all-objects --batch-check" && + midx_git_two_modes "cat-file --batch-all-objects --batch-check --unordered" sorted ' } From patchwork Thu Apr 4 23:22:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 10886635 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1538F139A for ; Thu, 4 Apr 2019 23:23:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 00DB428AAC for ; Thu, 4 Apr 2019 23:23:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E5BAE28ABB; Thu, 4 Apr 2019 23:23:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8CF3528AAC for ; Thu, 4 Apr 2019 23:23:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729703AbfDDXXB (ORCPT ); Thu, 4 Apr 2019 19:23:01 -0400 Received: from cloud.peff.net ([104.130.231.41]:47404 "HELO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1729501AbfDDXXB (ORCPT ); Thu, 4 Apr 2019 19:23:01 -0400 Received: (qmail 1416 invoked by uid 109); 4 Apr 2019 23:23:01 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with SMTP; Thu, 04 Apr 2019 23:23:01 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 20392 invoked by uid 111); 4 Apr 2019 23:23:27 -0000 Received: from sigill.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.7) by peff.net (qpsmtpd/0.94) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) SMTP; Thu, 04 Apr 2019 19:23:27 -0400 Authentication-Results: peff.net; auth=none Received: by sigill.intra.peff.net (sSMTP sendmail emulation); Thu, 04 Apr 2019 19:22:59 -0400 Date: Thu, 4 Apr 2019 19:22:59 -0400 From: Jeff King To: git@vger.kernel.org Subject: [PATCH 03/12] packfile: factor out .pack to .idx name conversion Message-ID: <20190404232259.GC21839@sigill.intra.peff.net> References: <20190404232104.GA27770@sigill.intra.peff.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190404232104.GA27770@sigill.intra.peff.net> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When we open a pack .idx, we have to convert the "foo.pack" name stored in the packed_git struct to "foo.idx". This isn't too complicated, but we do encode some policy in the form of a BUG(). Let's pull this into its own function so it can be reused. Signed-off-by: Jeff King --- packfile.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/packfile.c b/packfile.c index 16bcb75262..054269ae5d 100644 --- a/packfile.c +++ b/packfile.c @@ -186,18 +186,24 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map, return 0; } +static char *pack_name_to_idx(const char *pack_name) +{ + size_t len; + + if (!strip_suffix(pack_name, ".pack", &len)) + BUG("pack_name does not end in .pack"); + return xstrfmt("%.*s.idx", (int)len, pack_name); +} + int open_pack_index(struct packed_git *p) { char *idx_name; - size_t len; int ret; if (p->index_data) return 0; - if (!strip_suffix(p->pack_name, ".pack", &len)) - BUG("pack_name does not end in .pack"); - idx_name = xstrfmt("%.*s.idx", (int)len, p->pack_name); + idx_name = pack_name_to_idx(p->pack_name); ret = check_packed_git_idx(idx_name, p); free(idx_name); return ret; From patchwork Thu Apr 4 23:25:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 10886637 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7386A13B5 for ; Thu, 4 Apr 2019 23:25:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 50C21286E5 for ; Thu, 4 Apr 2019 23:25:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 40218286DD; Thu, 4 Apr 2019 23:25:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D148F286DD for ; Thu, 4 Apr 2019 23:25:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729212AbfDDXZs (ORCPT ); Thu, 4 Apr 2019 19:25:48 -0400 Received: from cloud.peff.net ([104.130.231.41]:47410 "HELO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1728211AbfDDXZs (ORCPT ); Thu, 4 Apr 2019 19:25:48 -0400 Received: (qmail 1448 invoked by uid 109); 4 Apr 2019 23:25:48 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with SMTP; Thu, 04 Apr 2019 23:25:48 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 20411 invoked by uid 111); 4 Apr 2019 23:26:15 -0000 Received: from sigill.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.7) by peff.net (qpsmtpd/0.94) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) SMTP; Thu, 04 Apr 2019 19:26:15 -0400 Authentication-Results: peff.net; auth=none Received: by sigill.intra.peff.net (sSMTP sendmail emulation); Thu, 04 Apr 2019 19:25:46 -0400 Date: Thu, 4 Apr 2019 19:25:46 -0400 From: Jeff King To: git@vger.kernel.org Subject: [PATCH 04/12] packfile: check midx coverage with .idx rather than .pack Message-ID: <20190404232546.GD21839@sigill.intra.peff.net> References: <20190404232104.GA27770@sigill.intra.peff.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190404232104.GA27770@sigill.intra.peff.net> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When we have a .midx that covers many packfiles, we try to avoid opening the .idx for those packfiles. However, there are a few problems with the filename comparison we use: - we ask midx_contains_pack() about the .pack name, not the .idx name. But it compares to the latter. - we compute the basename of the pack using strrchr() to find the final slash. But that leaves an extra "/" at the start of our string; we need to advance past it. That also raises the question of what to do when the name does not have a slash at all. This should generally not happen (we always find files in "pack/"), but it doesn't hurt to be defensive here. The tests don't notice because there's nothing about opening those .idx files that would cause us to give incorrect output. It's just a little slower. The new test checks this case by corrupting the covered .idx, and then making sure we don't complain about it. Signed-off-by: Jeff King --- packfile.c | 17 ++++++++++++++--- t/t5319-multi-pack-index.sh | 14 ++++++++++++++ 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/packfile.c b/packfile.c index 054269ae5d..e7ca135ed5 100644 --- a/packfile.c +++ b/packfile.c @@ -472,6 +472,16 @@ static unsigned int get_max_fd_limit(void) #endif } +static const char *pack_basename(struct packed_git *p) +{ + const char *ret = strrchr(p->pack_name, '/'); + if (ret) + ret = ret + 1; /* skip past slash */ + else + ret = p->pack_name; /* we only have a base */ + return ret; +} + /* * Do not call this directly as this leaks p->pack_fd on error return; * call open_packed_git() instead. @@ -486,15 +496,16 @@ static int open_packed_git_1(struct packed_git *p) ssize_t read_result; const unsigned hashsz = the_hash_algo->rawsz; - if (!p->index_data) { + if (!p->index_data && the_repository->objects->multi_pack_index) { struct multi_pack_index *m; - const char *pack_name = strrchr(p->pack_name, '/'); + char *idx_name = pack_name_to_idx(pack_basename(p)); for (m = the_repository->objects->multi_pack_index; m; m = m->next) { - if (midx_contains_pack(m, pack_name)) + if (midx_contains_pack(m, idx_name)) break; } + free(idx_name); if (!m && open_pack_index(p)) return error("packfile %s index unavailable", p->pack_name); diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 8c4d2bd849..1ebf19ec3c 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -117,6 +117,20 @@ test_expect_success 'write midx with one v2 pack' ' compare_results_with_midx "one v2 pack" +test_expect_success 'corrupt idx not opened' ' + idx=$(test-tool read-midx $objdir | grep "\.idx\$") && + mv $objdir/pack/$idx backup-$idx && + test_when_finished "mv backup-\$idx \$objdir/pack/\$idx" && + + # This is the minimum size for a sha-1 based .idx; this lets + # us pass perfunctory tests, but anything that actually opens and reads + # the idx file will complain. + test_copy_bytes 1064 $objdir/pack/$idx && + + git -c core.multiPackIndex=true rev-list --objects --all 2>err && + test_must_be_empty err +' + test_expect_success 'add more objects' ' for i in $(test_seq 6 10) do From patchwork Thu Apr 4 23:27:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 10886645 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5060013B5 for ; Thu, 4 Apr 2019 23:27:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 39A39286DD for ; Thu, 4 Apr 2019 23:27:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2C97E28ABE; Thu, 4 Apr 2019 23:27:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C0A31286DD for ; Thu, 4 Apr 2019 23:27:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729594AbfDDX1G (ORCPT ); Thu, 4 Apr 2019 19:27:06 -0400 Received: from cloud.peff.net ([104.130.231.41]:47416 "HELO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1729251AbfDDX1G (ORCPT ); Thu, 4 Apr 2019 19:27:06 -0400 Received: (qmail 1467 invoked by uid 109); 4 Apr 2019 23:27:06 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with SMTP; Thu, 04 Apr 2019 23:27:06 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 20429 invoked by uid 111); 4 Apr 2019 23:27:32 -0000 Received: from sigill.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.7) by peff.net (qpsmtpd/0.94) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) SMTP; Thu, 04 Apr 2019 19:27:32 -0400 Authentication-Results: peff.net; auth=none Received: by sigill.intra.peff.net (sSMTP sendmail emulation); Thu, 04 Apr 2019 19:27:04 -0400 Date: Thu, 4 Apr 2019 19:27:04 -0400 From: Jeff King To: git@vger.kernel.org Subject: [PATCH 05/12] http: simplify parsing of remote objects/info/packs Message-ID: <20190404232704.GE21839@sigill.intra.peff.net> References: <20190404232104.GA27770@sigill.intra.peff.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190404232104.GA27770@sigill.intra.peff.net> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We can use skip_prefix() and parse_oid_hex() to continuously increment our pointer, rather than dealing with magic numbers. This also fixes a few small shortcomings: - if we see a 'P' line that does not match our expectations, we'll leave our "i" counter in the middle of the line. So we'll parse: "P P P pack-1234.pack" as if there was just one "P" which was not intentional (though probably not that harmful). - if we see a line with the right prefix, suffix, and length, i.e. matching /P pack-.{40}.pack\n/, we'll interpret the middle part as hex without checking if it could be parsed. This could lead to us looking at uninitialized garbage in the hash array. In practice this means we'll just make a garbage request to the server which will fail, though it's interesting that a malicious server could convince us to leak 40 bytes of uninitialized stack to them. - the current code is picky about seeing a newline at the end of file, but we can easily be more liberal Signed-off-by: Jeff King --- http.c | 35 ++++++++++++++--------------------- 1 file changed, 14 insertions(+), 21 deletions(-) diff --git a/http.c b/http.c index a32ad36ddf..2ef47bc779 100644 --- a/http.c +++ b/http.c @@ -2147,11 +2147,11 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head, int http_get_info_packs(const char *base_url, struct packed_git **packs_head) { struct http_get_options options = {0}; - int ret = 0, i = 0; - char *url, *data; + int ret = 0; + char *url; + const char *data; struct strbuf buf = STRBUF_INIT; - unsigned char hash[GIT_MAX_RAWSZ]; - const unsigned hexsz = the_hash_algo->hexsz; + struct object_id oid; end_url_with_slash(&buf, base_url); strbuf_addstr(&buf, "objects/info/packs"); @@ -2163,24 +2163,17 @@ int http_get_info_packs(const char *base_url, struct packed_git **packs_head) goto cleanup; data = buf.buf; - while (i < buf.len) { - switch (data[i]) { - case 'P': - i++; - if (i + hexsz + 12 <= buf.len && - starts_with(data + i, " pack-") && - starts_with(data + i + hexsz + 6, ".pack\n")) { - get_sha1_hex(data + i + 6, hash); - fetch_and_setup_pack_index(packs_head, hash, - base_url); - i += hexsz + 11; - break; - } - default: - while (i < buf.len && data[i] != '\n') - i++; + while (*data) { + if (skip_prefix(data, "P pack-", &data) && + !parse_oid_hex(data, &oid, &data) && + skip_prefix(data, ".pack", &data) && + (*data == '\n' || *data == '\0')) { + fetch_and_setup_pack_index(packs_head, oid.hash, base_url); + } else { + data = strchrnul(data, '\n'); } - i++; + if (*data) + data++; /* skip past newline */ } cleanup: From patchwork Thu Apr 4 23:27:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 10886647 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 39DFB139A for ; Thu, 4 Apr 2019 23:27:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 24AD0286E5 for ; Thu, 4 Apr 2019 23:27:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 184F328AC2; Thu, 4 Apr 2019 23:27:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A8DDD286E5 for ; Thu, 4 Apr 2019 23:27:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729961AbfDDX1p (ORCPT ); Thu, 4 Apr 2019 19:27:45 -0400 Received: from cloud.peff.net ([104.130.231.41]:47422 "HELO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1727039AbfDDX1p (ORCPT ); Thu, 4 Apr 2019 19:27:45 -0400 Received: (qmail 1482 invoked by uid 109); 4 Apr 2019 23:27:45 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with SMTP; Thu, 04 Apr 2019 23:27:45 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 20445 invoked by uid 111); 4 Apr 2019 23:28:12 -0000 Received: from sigill.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.7) by peff.net (qpsmtpd/0.94) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) SMTP; Thu, 04 Apr 2019 19:28:12 -0400 Authentication-Results: peff.net; auth=none Received: by sigill.intra.peff.net (sSMTP sendmail emulation); Thu, 04 Apr 2019 19:27:43 -0400 Date: Thu, 4 Apr 2019 19:27:43 -0400 From: Jeff King To: git@vger.kernel.org Subject: [PATCH 06/12] server-info: fix blind pointer arithmetic Message-ID: <20190404232743.GF21839@sigill.intra.peff.net> References: <20190404232104.GA27770@sigill.intra.peff.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190404232104.GA27770@sigill.intra.peff.net> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When we're writing out a new objects/info/packs file, we read back the old one to try to keep the ordering the same. When we see a line starting with "P", we expect "P pack-1234..." and blindly jump to "line + 2" to parse the pack name. If we saw a line with _just_ "P" and nothing else, we'd jump past the end of the buffer and start reading arbitrary memory. This shouldn't be a big attack vector, as the files are local to the repository and written by us, but it's clearly worth fixing (we do read remote copies of the file for dumb-http fetches, but using a totally different parser!). Let's instead use skip_prefix() here, which avoids pointer arithmetic altogether. Note that this converts our switch statement to an if/else chain, making it slightly more verbose. But it will also make it easier to do a few follow-on cleanups. Signed-off-by: Jeff King --- server-info.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/server-info.c b/server-info.c index e2b2d6a27a..b61a6be4c2 100644 --- a/server-info.c +++ b/server-info.c @@ -112,9 +112,9 @@ static struct pack_info *find_pack_by_name(const char *name) /* Returns non-zero when we detect that the info in the * old file is useless. */ -static int parse_pack_def(const char *line, int old_cnt) +static int parse_pack_def(const char *packname, int old_cnt) { - struct pack_info *i = find_pack_by_name(line + 2); + struct pack_info *i = find_pack_by_name(packname); if (i) { i->old_num = old_cnt; return 0; @@ -139,24 +139,26 @@ static int read_pack_info_file(const char *infofile) return 1; /* nonexistent is not an error. */ while (fgets(line, sizeof(line), fp)) { + const char *arg; int len = strlen(line); if (len && line[len-1] == '\n') line[--len] = 0; if (!len) continue; - switch (line[0]) { - case 'P': /* P name */ - if (parse_pack_def(line, old_cnt++)) + if (skip_prefix(line, "P ", &arg)) { + /* P name */ + if (parse_pack_def(arg, old_cnt++)) goto out_stale; - break; - case 'D': /* we used to emit D but that was misguided. */ - case 'T': /* we used to emit T but nobody uses it. */ + } else if (line[0] == 'D') { + /* we used to emit D but that was misguided. */ goto out_stale; - default: + } else if (line[0] == 'T') { + /* we used to emit T but nobody uses it. */ + goto out_stale; + } else { error("unrecognized: %s", line); - break; } } fclose(fp); From patchwork Thu Apr 4 23:28:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 10886649 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B7ECB17EE for ; Thu, 4 Apr 2019 23:28:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A0E8E286E5 for ; Thu, 4 Apr 2019 23:28:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9200E28AC2; Thu, 4 Apr 2019 23:28:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 15FF9286E5 for ; Thu, 4 Apr 2019 23:28:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729967AbfDDX2L (ORCPT ); Thu, 4 Apr 2019 19:28:11 -0400 Received: from cloud.peff.net ([104.130.231.41]:47428 "HELO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1727039AbfDDX2K (ORCPT ); Thu, 4 Apr 2019 19:28:10 -0400 Received: (qmail 1489 invoked by uid 109); 4 Apr 2019 23:28:11 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with SMTP; Thu, 04 Apr 2019 23:28:11 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 20461 invoked by uid 111); 4 Apr 2019 23:28:37 -0000 Received: from sigill.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.7) by peff.net (qpsmtpd/0.94) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) SMTP; Thu, 04 Apr 2019 19:28:37 -0400 Authentication-Results: peff.net; auth=none Received: by sigill.intra.peff.net (sSMTP sendmail emulation); Thu, 04 Apr 2019 19:28:08 -0400 Date: Thu, 4 Apr 2019 19:28:08 -0400 From: Jeff King To: git@vger.kernel.org Subject: [PATCH 07/12] server-info: simplify cleanup in parse_pack_def() Message-ID: <20190404232808.GG21839@sigill.intra.peff.net> References: <20190404232104.GA27770@sigill.intra.peff.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190404232104.GA27770@sigill.intra.peff.net> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We have two exits from the function: either we jump to the out_stale label or not. But in both exits we repeat our cleanup, and the only difference is our return value. Let's just use a variable for the return value to avoid repeating ourselves. Signed-off-by: Jeff King --- server-info.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/server-info.c b/server-info.c index b61a6be4c2..ba44cece7f 100644 --- a/server-info.c +++ b/server-info.c @@ -133,6 +133,7 @@ static int read_pack_info_file(const char *infofile) FILE *fp; char line[1000]; int old_cnt = 0; + int stale = 1; fp = fopen_or_warn(infofile, "r"); if (!fp) @@ -161,11 +162,11 @@ static int read_pack_info_file(const char *infofile) error("unrecognized: %s", line); } } - fclose(fp); - return 0; + stale = 0; + out_stale: fclose(fp); - return 1; + return stale; } static int compare_info(const void *a_, const void *b_) From patchwork Thu Apr 4 23:28:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 10886651 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2124C13B5 for ; Thu, 4 Apr 2019 23:28:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0D6BD28AC2 for ; Thu, 4 Apr 2019 23:28:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0204C28AD2; Thu, 4 Apr 2019 23:28:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9B62D28AC2 for ; Thu, 4 Apr 2019 23:28:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729997AbfDDX2o (ORCPT ); Thu, 4 Apr 2019 19:28:44 -0400 Received: from cloud.peff.net ([104.130.231.41]:47434 "HELO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1727039AbfDDX2o (ORCPT ); Thu, 4 Apr 2019 19:28:44 -0400 Received: (qmail 1501 invoked by uid 109); 4 Apr 2019 23:28:44 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with SMTP; Thu, 04 Apr 2019 23:28:44 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 20477 invoked by uid 111); 4 Apr 2019 23:29:10 -0000 Received: from sigill.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.7) by peff.net (qpsmtpd/0.94) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) SMTP; Thu, 04 Apr 2019 19:29:10 -0400 Authentication-Results: peff.net; auth=none Received: by sigill.intra.peff.net (sSMTP sendmail emulation); Thu, 04 Apr 2019 19:28:42 -0400 Date: Thu, 4 Apr 2019 19:28:42 -0400 From: Jeff King To: git@vger.kernel.org Subject: [PATCH 08/12] server-info: use strbuf to read old info/packs file Message-ID: <20190404232842.GH21839@sigill.intra.peff.net> References: <20190404232104.GA27770@sigill.intra.peff.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190404232104.GA27770@sigill.intra.peff.net> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This old code uses fgets with a fixed-size buffer. Let's use a strbuf instead, so we don't have to wonder if "1000" is big enough, or what happens if we see a long line. This also lets us drop our custom code to trim the newline. Probably nobody actually cares about the 1000-char limit (after all, the lines generally only say "P pack-[0-9a-f]{40}.pack"), so this is mostly just about cleanup/readability. Signed-off-by: Jeff King --- server-info.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/server-info.c b/server-info.c index ba44cece7f..d4115fecbb 100644 --- a/server-info.c +++ b/server-info.c @@ -131,40 +131,38 @@ static int parse_pack_def(const char *packname, int old_cnt) static int read_pack_info_file(const char *infofile) { FILE *fp; - char line[1000]; + struct strbuf line = STRBUF_INIT; int old_cnt = 0; int stale = 1; fp = fopen_or_warn(infofile, "r"); if (!fp) return 1; /* nonexistent is not an error. */ - while (fgets(line, sizeof(line), fp)) { + while (strbuf_getline(&line, fp) != EOF) { const char *arg; - int len = strlen(line); - if (len && line[len-1] == '\n') - line[--len] = 0; - if (!len) + if (!line.len) continue; - if (skip_prefix(line, "P ", &arg)) { + if (skip_prefix(line.buf, "P ", &arg)) { /* P name */ if (parse_pack_def(arg, old_cnt++)) goto out_stale; - } else if (line[0] == 'D') { + } else if (line.buf[0] == 'D') { /* we used to emit D but that was misguided. */ goto out_stale; - } else if (line[0] == 'T') { + } else if (line.buf[0] == 'T') { /* we used to emit T but nobody uses it. */ goto out_stale; } else { - error("unrecognized: %s", line); + error("unrecognized: %s", line.buf); } } stale = 0; out_stale: + strbuf_release(&line); fclose(fp); return stale; } From patchwork Thu Apr 4 23:29:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 10886653 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 38578139A for ; Thu, 4 Apr 2019 23:29:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 24A1728AC2 for ; Thu, 4 Apr 2019 23:29:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 18D1828AF2; Thu, 4 Apr 2019 23:29:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C09F228AC2 for ; Thu, 4 Apr 2019 23:29:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730026AbfDDX3P (ORCPT ); Thu, 4 Apr 2019 19:29:15 -0400 Received: from cloud.peff.net ([104.130.231.41]:47440 "HELO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1729999AbfDDX3P (ORCPT ); Thu, 4 Apr 2019 19:29:15 -0400 Received: (qmail 1513 invoked by uid 109); 4 Apr 2019 23:29:15 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with SMTP; Thu, 04 Apr 2019 23:29:15 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 20497 invoked by uid 111); 4 Apr 2019 23:29:42 -0000 Received: from sigill.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.7) by peff.net (qpsmtpd/0.94) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) SMTP; Thu, 04 Apr 2019 19:29:42 -0400 Authentication-Results: peff.net; auth=none Received: by sigill.intra.peff.net (sSMTP sendmail emulation); Thu, 04 Apr 2019 19:29:13 -0400 Date: Thu, 4 Apr 2019 19:29:13 -0400 From: Jeff King To: git@vger.kernel.org Subject: [PATCH 09/12] server-info: drop nr_alloc struct member Message-ID: <20190404232913.GI21839@sigill.intra.peff.net> References: <20190404232104.GA27770@sigill.intra.peff.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190404232104.GA27770@sigill.intra.peff.net> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We keep an array of struct pointers, with each one representing a single packfile. But for some reason there is a nr_alloc parameter inside each struct, which has never been used. This is probably cruft left over from development, where we might have wanted a nr_alloc to dynamically grow the list. But as it turns out, we do not dynamically grow the list at all, but rather count up the total number of packs and use that as a maximum size. So while we're thinking of this, let's add an assert() that documents (and checks!) that our allocation and fill loops stay in sync. Signed-off-by: Jeff King --- server-info.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server-info.c b/server-info.c index d4115fecbb..c9fbfd3a51 100644 --- a/server-info.c +++ b/server-info.c @@ -91,7 +91,6 @@ static struct pack_info { struct packed_git *p; int old_num; int new_num; - int nr_alloc; } **info; static int num_pack; static const char *objdir; @@ -213,6 +212,7 @@ static void init_pack_info(const char *infofile, int force) for (i = 0, p = get_all_packs(the_repository); p; p = p->next) { if (!p->pack_local) continue; + assert(i < num_pack); info[i] = xcalloc(1, sizeof(struct pack_info)); info[i]->p = p; info[i]->old_num = -1; From patchwork Thu Apr 4 23:30:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 10886657 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E07EC15AC for ; Thu, 4 Apr 2019 23:30:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C5B8228B00 for ; Thu, 4 Apr 2019 23:30:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B454528B03; Thu, 4 Apr 2019 23:30:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E456128B00 for ; Thu, 4 Apr 2019 23:30:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730371AbfDDXaL (ORCPT ); Thu, 4 Apr 2019 19:30:11 -0400 Received: from cloud.peff.net ([104.130.231.41]:47450 "HELO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1729267AbfDDXaL (ORCPT ); Thu, 4 Apr 2019 19:30:11 -0400 Received: (qmail 1532 invoked by uid 109); 4 Apr 2019 23:30:10 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with SMTP; Thu, 04 Apr 2019 23:30:10 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 20531 invoked by uid 111); 4 Apr 2019 23:30:37 -0000 Received: from sigill.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.7) by peff.net (qpsmtpd/0.94) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) SMTP; Thu, 04 Apr 2019 19:30:37 -0400 Authentication-Results: peff.net; auth=none Received: by sigill.intra.peff.net (sSMTP sendmail emulation); Thu, 04 Apr 2019 19:30:08 -0400 Date: Thu, 4 Apr 2019 19:30:08 -0400 From: Jeff King To: git@vger.kernel.org Subject: [PATCH 10/12] packfile.h: drop extern from function declarations Message-ID: <20190404233008.GJ21839@sigill.intra.peff.net> References: <20190404232104.GA27770@sigill.intra.peff.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190404232104.GA27770@sigill.intra.peff.net> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP As CodingGuidelines recommends, we do not need an "extern" when declaring a public function. Let's drop these. Note that we leave the extern on report_garbage(), as that is actually a function pointer, not a function itself. Signed-off-by: Jeff King --- I hesitated on this one, because it does have a minor conflict with pu. But I just couldn't bring myself to add a new extern, nor to stick a non-extern one amidst a bunch of others. packfile.h | 80 +++++++++++++++++++++++++++--------------------------- 1 file changed, 40 insertions(+), 40 deletions(-) diff --git a/packfile.h b/packfile.h index d70c6d9afb..b40fc34fb2 100644 --- a/packfile.h +++ b/packfile.h @@ -15,23 +15,23 @@ struct object_info; * * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" */ -extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext); +char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext); /* * Return the name of the (local) packfile with the specified sha1 in * its name. The return value is a pointer to memory that is * overwritten each time this function is called. */ -extern char *sha1_pack_name(const unsigned char *sha1); +char *sha1_pack_name(const unsigned char *sha1); /* * Return the name of the (local) pack index file with the specified * sha1 in its name. The return value is a pointer to memory that is * overwritten each time this function is called. */ -extern char *sha1_pack_index_name(const unsigned char *sha1); +char *sha1_pack_index_name(const unsigned char *sha1); -extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path); +struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path); typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len, const char *file_pach, void *data); @@ -45,8 +45,8 @@ void for_each_file_in_pack_dir(const char *objdir, #define PACKDIR_FILE_GARBAGE 4 extern void (*report_garbage)(unsigned seen_bits, const char *path); -extern void reprepare_packed_git(struct repository *r); -extern void install_packed_git(struct repository *r, struct packed_git *pack); +void reprepare_packed_git(struct repository *r); +void install_packed_git(struct repository *r, struct packed_git *pack); struct packed_git *get_packed_git(struct repository *r); struct list_head *get_packed_git_mru(struct repository *r); @@ -59,32 +59,32 @@ struct packed_git *get_all_packs(struct repository *r); */ unsigned long approximate_object_count(void); -extern struct packed_git *find_sha1_pack(const unsigned char *sha1, - struct packed_git *packs); +struct packed_git *find_sha1_pack(const unsigned char *sha1, + struct packed_git *packs); -extern void pack_report(void); +void pack_report(void); /* * mmap the index file for the specified packfile (if it is not * already mmapped). Return 0 on success. */ -extern int open_pack_index(struct packed_git *); +int open_pack_index(struct packed_git *); /* * munmap the index file for the specified packfile (if it is * currently mmapped). */ -extern void close_pack_index(struct packed_git *); +void close_pack_index(struct packed_git *); -extern uint32_t get_pack_fanout(struct packed_git *p, uint32_t value); +uint32_t get_pack_fanout(struct packed_git *p, uint32_t value); -extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *); -extern void close_pack_windows(struct packed_git *); -extern void close_pack(struct packed_git *); -extern void close_all_packs(struct raw_object_store *o); -extern void unuse_pack(struct pack_window **); -extern void clear_delta_base_cache(void); -extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local); +unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *); +void close_pack_windows(struct packed_git *); +void close_pack(struct packed_git *); +void close_all_packs(struct raw_object_store *o); +void unuse_pack(struct pack_window **); +void clear_delta_base_cache(void); +struct packed_git *add_packed_git(const char *path, size_t path_len, int local); /* * Make sure that a pointer access into an mmap'd index file is within bounds, @@ -94,7 +94,7 @@ extern struct packed_git *add_packed_git(const char *path, size_t path_len, int * (like the 64-bit extended offset table), as we compare the size to the * fixed-length parts when we open the file. */ -extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr); +void check_pack_index_ptr(const struct packed_git *p, const void *ptr); /* * Perform binary search on a pack-index for a given oid. Packfile is expected to @@ -110,59 +110,59 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32 * at the SHA-1 within the mmapped index. Return NULL if there is an * error. */ -extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t n); +const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t n); /* * Like nth_packed_object_sha1, but write the data into the object specified by * the the first argument. Returns the first argument on success, and NULL on * error. */ -extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n); +const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n); /* * Return the offset of the nth object within the specified packfile. * The index must already be opened. */ -extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n); +off_t nth_packed_object_offset(const struct packed_git *, uint32_t n); /* * If the object named sha1 is present in the specified packfile, * return its offset within the packfile; otherwise, return 0. */ -extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *); +off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *); -extern int is_pack_valid(struct packed_git *); -extern void *unpack_entry(struct repository *r, struct packed_git *, off_t, enum object_type *, unsigned long *); -extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep); -extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t); -extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *); +int is_pack_valid(struct packed_git *); +void *unpack_entry(struct repository *r, struct packed_git *, off_t, enum object_type *, unsigned long *); +unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep); +unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t); +int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *); -extern void release_pack_memory(size_t); +void release_pack_memory(size_t); /* global flag to enable extra checks when accessing packed objects */ -extern int do_check_packed_object_crc; +int do_check_packed_object_crc; -extern int packed_object_info(struct repository *r, +int packed_object_info(struct repository *r, struct packed_git *pack, off_t offset, struct object_info *); -extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1); -extern const struct packed_git *has_packed_and_bad(struct repository *r, const unsigned char *sha1); +void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1); +const struct packed_git *has_packed_and_bad(struct repository *r, const unsigned char *sha1); /* * Iff a pack file in the given repository contains the object named by sha1, * return true and store its location to e. */ -extern int find_pack_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e); +int find_pack_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e); -extern int has_object_pack(const struct object_id *oid); +int has_object_pack(const struct object_id *oid); -extern int has_pack_index(const unsigned char *sha1); +int has_pack_index(const unsigned char *sha1); /* * Return 1 if an object in a promisor packfile is or refers to the given * object, 0 otherwise. */ -extern int is_promisor_object(const struct object_id *oid); +int is_promisor_object(const struct object_id *oid); /* * Expose a function for fuzz testing. @@ -174,7 +174,7 @@ extern int is_promisor_object(const struct object_id *oid); * have a convenient entry-point for fuzz testing. For real uses, you should * probably use open_pack_index() or parse_pack_index() instead. */ -extern int load_idx(const char *path, const unsigned int hashsz, void *idx_map, - size_t idx_size, struct packed_git *p); +int load_idx(const char *path, const unsigned int hashsz, void *idx_map, + size_t idx_size, struct packed_git *p); #endif From patchwork Thu Apr 4 23:30:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 10886659 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4DB3B15AC for ; Thu, 4 Apr 2019 23:30:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3798E28B00 for ; Thu, 4 Apr 2019 23:30:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2AA6B28B03; Thu, 4 Apr 2019 23:30:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B8AB728B00 for ; Thu, 4 Apr 2019 23:30:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730389AbfDDXau (ORCPT ); Thu, 4 Apr 2019 19:30:50 -0400 Received: from cloud.peff.net ([104.130.231.41]:47456 "HELO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1729999AbfDDXau (ORCPT ); Thu, 4 Apr 2019 19:30:50 -0400 Received: (qmail 1543 invoked by uid 109); 4 Apr 2019 23:30:50 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with SMTP; Thu, 04 Apr 2019 23:30:50 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 20547 invoked by uid 111); 4 Apr 2019 23:31:17 -0000 Received: from sigill.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.7) by peff.net (qpsmtpd/0.94) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) SMTP; Thu, 04 Apr 2019 19:31:17 -0400 Authentication-Results: peff.net; auth=none Received: by sigill.intra.peff.net (sSMTP sendmail emulation); Thu, 04 Apr 2019 19:30:48 -0400 Date: Thu, 4 Apr 2019 19:30:48 -0400 From: Jeff King To: git@vger.kernel.org Subject: [PATCH 11/12] server-info: drop objdirlen pointer arithmetic Message-ID: <20190404233048.GK21839@sigill.intra.peff.net> References: <20190404232104.GA27770@sigill.intra.peff.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190404232104.GA27770@sigill.intra.peff.net> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When writing objects/info/packs, we use the basename of each pack (i.e., just the "pack-1234abcd.pack" part). We compute that manually by adding "objdirlen + 6" to the name. This _should_ work consistently, as we do not include non-local packs, meaning everything should be in $objdir/pack/. Before f13d7db4af (server-info.c: use pack_local like everybody else., 2005-12-05), this was definitely true, since we computed "local" based on comparing the objdir string. Since then, we're relying on the code on packfile.c to match our expectations of p->pack_name and p->local. I think our expectations do still hold today, but we can be a bit more defensive by just using pack_basename() to get the base. That future-proofs us, and should hopefully be more obviously safe to somebody reading the code. Signed-off-by: Jeff King --- packfile.c | 2 +- packfile.h | 6 ++++++ server-info.c | 10 ++-------- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/packfile.c b/packfile.c index e7ca135ed5..a0f92b7eef 100644 --- a/packfile.c +++ b/packfile.c @@ -472,7 +472,7 @@ static unsigned int get_max_fd_limit(void) #endif } -static const char *pack_basename(struct packed_git *p) +const char *pack_basename(struct packed_git *p) { const char *ret = strrchr(p->pack_name, '/'); if (ret) diff --git a/packfile.h b/packfile.h index b40fc34fb2..45bf792d79 100644 --- a/packfile.h +++ b/packfile.h @@ -31,6 +31,12 @@ char *sha1_pack_name(const unsigned char *sha1); */ char *sha1_pack_index_name(const unsigned char *sha1); +/* + * Return the basename of the packfile, omitting any containing directory + * (e.g., "pack-1234abcd[...].pack"). + */ +const char *pack_basename(struct packed_git *p); + struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path); typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len, diff --git a/server-info.c b/server-info.c index c9fbfd3a51..ab03c1b3c2 100644 --- a/server-info.c +++ b/server-info.c @@ -93,16 +93,13 @@ static struct pack_info { int new_num; } **info; static int num_pack; -static const char *objdir; -static int objdirlen; static struct pack_info *find_pack_by_name(const char *name) { int i; for (i = 0; i < num_pack; i++) { struct packed_git *p = info[i]->p; - /* skip "/pack/" after ".git/objects" */ - if (!strcmp(p->pack_name + objdirlen + 6, name)) + if (!strcmp(pack_basename(p), name)) return info[i]; } return NULL; @@ -196,9 +193,6 @@ static void init_pack_info(const char *infofile, int force) int stale; int i = 0; - objdir = get_object_directory(); - objdirlen = strlen(objdir); - for (p = get_all_packs(the_repository); p; p = p->next) { /* we ignore things on alternate path since they are * not available to the pullers in general. @@ -246,7 +240,7 @@ static int write_pack_info_file(FILE *fp) { int i; for (i = 0; i < num_pack; i++) { - if (fprintf(fp, "P %s\n", info[i]->p->pack_name + objdirlen + 6) < 0) + if (fprintf(fp, "P %s\n", pack_basename(info[i]->p)) < 0) return -1; } if (fputc('\n', fp) == EOF) From patchwork Thu Apr 4 23:31:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 10886661 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3C3D715AC for ; Thu, 4 Apr 2019 23:31:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 288D728B00 for ; Thu, 4 Apr 2019 23:31:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1CEBA28B03; Thu, 4 Apr 2019 23:31:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A99E428B00 for ; Thu, 4 Apr 2019 23:31:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729999AbfDDXbH (ORCPT ); Thu, 4 Apr 2019 19:31:07 -0400 Received: from cloud.peff.net ([104.130.231.41]:47462 "HELO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1729098AbfDDXbH (ORCPT ); Thu, 4 Apr 2019 19:31:07 -0400 Received: (qmail 1551 invoked by uid 109); 4 Apr 2019 23:31:07 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with SMTP; Thu, 04 Apr 2019 23:31:07 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 20563 invoked by uid 111); 4 Apr 2019 23:31:34 -0000 Received: from sigill.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.7) by peff.net (qpsmtpd/0.94) with (ECDHE-RSA-AES256-GCM-SHA384 encrypted) SMTP; Thu, 04 Apr 2019 19:31:34 -0400 Authentication-Results: peff.net; auth=none Received: by sigill.intra.peff.net (sSMTP sendmail emulation); Thu, 04 Apr 2019 19:31:05 -0400 Date: Thu, 4 Apr 2019 19:31:05 -0400 From: Jeff King To: git@vger.kernel.org Subject: [PATCH 12/12] update_info_refs(): drop unused force parameter Message-ID: <20190404233105.GL21839@sigill.intra.peff.net> References: <20190404232104.GA27770@sigill.intra.peff.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190404232104.GA27770@sigill.intra.peff.net> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Once upon a time the force flag meant something when writing info/refs, but it hasn't done anything since 60d0526aaa (Unoptimize info/refs creation., 2005-09-14). Signed-off-by: Jeff King --- server-info.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/server-info.c b/server-info.c index ab03c1b3c2..41274d098b 100644 --- a/server-info.c +++ b/server-info.c @@ -78,7 +78,7 @@ static int generate_info_refs(FILE *fp) return for_each_ref(add_info_ref, fp); } -static int update_info_refs(int force) +static int update_info_refs(void) { char *path = git_pathdup("info/refs"); int ret = update_info_file(path, generate_info_refs); @@ -269,7 +269,7 @@ int update_server_info(int force) */ int errs = 0; - errs = errs | update_info_refs(force); + errs = errs | update_info_refs(); errs = errs | update_info_packs(force); /* remove leftover rev-cache file if there is any */