From patchwork Fri Jun 24 05:27:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Han Xin X-Patchwork-Id: 12893888 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EE5FC433EF for ; Fri, 24 Jun 2022 05:28:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231461AbiFXF2z (ORCPT ); Fri, 24 Jun 2022 01:28:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229643AbiFXF2u (ORCPT ); Fri, 24 Jun 2022 01:28:50 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3029C67E7F for ; Thu, 23 Jun 2022 22:28:50 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id l6so1145693plg.11 for ; Thu, 23 Jun 2022 22:28:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ChF47S2kSfBb6+MelZOzdf0mfG/imwAGHvP5siEpmz0=; b=5hoqiE8IhCyxiznMFkIspnphAPQSHuZYMd9bzhAv5HNio/9+dBCvXTq96dZLo8I6lz EBMwAdch42n/QYVK/ubSXRXF1/XzH5GcGIadhTos2xR7+ccy4dBBwbnDt9JfcmB6eAjh t7BTDhMxfnv1ELtwAR0IsNnrbyzsGm9C9UDlu38vq9s8rOk2iZu8MPeJgQ3dPpHh7x1s pzKUlo193EaSIGg5a5yQWZYzt10/5PYMszsMQejNg3rge4mNADsXSUU2O4rdZcQWCBCW 042hfI/u3ZRhXkZQTEXLI+Li6JX07++KVi7pO0oitmyMy/cbvHkfP20D74WZEpojAton RIIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ChF47S2kSfBb6+MelZOzdf0mfG/imwAGHvP5siEpmz0=; b=N4dCW8SwTZWkQW1S8MDxKGp7HLt/n9ecXPJwetNgdJTkXK1T5V9cHLNi8oVSxQVAbP qi/uJiUosWOaPI1JlITS4Jb9672FI0c09/+AxfGemYrLJ3W2XmN4coCfQhd+QWeRP6mB jHo/DY5ZKP+MvkKFNkwOp/vf4xQ07G3wsm/xD5nne5ij3sFUxJWvY0TvUkv2xElzMyet mohS4SPBsEp0DFEOCxEt033fzneL72WTiCzewkXuC4uEsYjmI8cQJs4jkFYKT6UUdMgI JQs69EYO64b0z2eWMZN2GblqhEO1lQ+NRYcxLNpD8k41FsFVkTHqog9JBsUA83Pzl2Aq F0nw== X-Gm-Message-State: AJIora8JLLHkjVF2imHuKvmNsG8e8utV4iSKA44/vv+DdbYm6Ai/UP9T W0UfUoVYwgw/3/cU/4vaYJlbmA== X-Google-Smtp-Source: AGRyM1s4UtTWMgxc16Uc3gp2sagXyasxNvEEWFVVko5yZbFOTSU+0kvWMnzz7cKJauuPoaua6NoPjA== X-Received: by 2002:a17:90b:4c49:b0:1ec:e8a2:b5f0 with SMTP id np9-20020a17090b4c4900b001ece8a2b5f0mr1886008pjb.21.1656048529694; Thu, 23 Jun 2022 22:28:49 -0700 (PDT) Received: from JMHNXMC7VH.bytedance.net ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id s144-20020a632c96000000b003c265b7d4f6sm602960pgs.44.2022.06.23.22.28.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 23 Jun 2022 22:28:49 -0700 (PDT) From: Han Xin To: hanxin.hx@bytedance.com Cc: chiyutianyi@gmail.com, derrickstolee@github.com, git@vger.kernel.org, haiyangtand@gmail.com, jonathantanmy@google.com, me@ttaylorr.com, ps@pks.im Subject: [PATCH v2 1/2] test-lib.sh: add limited processes to test-lib Date: Fri, 24 Jun 2022 13:27:56 +0800 Message-Id: <442a4c351dea603e226bae89eddc2b3496d93262.1656044659.git.hanxin.hx@bytedance.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: <20220618030130.36419-1-hanxin.hx@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org We will use the lazy prerequisite ULIMIT_PROCESSES in a follow-up commit. With run_with_limited_processses() we can limit forking subprocesses and fail reliably in some test cases. Signed-off-by: Han Xin --- t/test-lib.sh | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/t/test-lib.sh b/t/test-lib.sh index 8ba5ca1534..f920e3b0ae 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1816,6 +1816,15 @@ test_lazy_prereq ULIMIT_FILE_DESCRIPTORS ' run_with_limited_open_files true ' +run_with_limited_processses () { + (ulimit -u 512 && "$@") +} + +test_lazy_prereq ULIMIT_PROCESSES ' + test_have_prereq !HPPA,!MINGW,!CYGWIN && + run_with_limited_processses true +' + build_option () { git version --build-options | sed -ne "s/^$1: //p" From patchwork Fri Jun 24 05:27:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Han Xin X-Patchwork-Id: 12893889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25631C433EF for ; Fri, 24 Jun 2022 05:29:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231297AbiFXF26 (ORCPT ); Fri, 24 Jun 2022 01:28:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53660 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231446AbiFXF2z (ORCPT ); Fri, 24 Jun 2022 01:28:55 -0400 Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 41ED568034 for ; Thu, 23 Jun 2022 22:28:54 -0700 (PDT) Received: by mail-pl1-x634.google.com with SMTP id o18so1188234plg.2 for ; Thu, 23 Jun 2022 22:28:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tvDbpF3jl9jqRJ7Uj9hcAM4/uPMrWiVTqy6urbJ5Ue0=; b=z2n1FuFaX/+1a+GFEOwBuleo+3ul402+BVHdpvQXpKjm1zw+fHRL52EZiGlHd/ZiQ3 Cc7+9IqTGmZL5ba8SLwc4wM0E2du57msHFdxLzo/ayDiUrbgqOiP9izrSBfMKm6DTGsa /qOjEoXYofGqPEs34OqPqCQGrkV7iXI/YUl5s0yMESZMghrwWX8OHM6P6MZVP/bHoMll DlobOMLQbeNf5HndGNj+iHe+2QKvoqAfpXNV2uGWi8zp6dYydoPWD4f2BHtUnI7bULvh fXCVTbtNX1WIYLtmiWt8Ovy0tyYxZBZ4ORsuozdIbcX9hG2o7Et5D3a4RCJ7Gflp+YS5 eapA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tvDbpF3jl9jqRJ7Uj9hcAM4/uPMrWiVTqy6urbJ5Ue0=; b=UGv5SDSVjVxUWF6TjLFsoY/wQZD79BOnpKgZULzZiV4nCvtpBJSTp+tBRfO+S1YATt l3EYJQD7nKfWqCW0AsqutptTM2uTmorEAcRkD8sVXtgFq4tG5/Aaatj0xcxfHIrIMYIJ XNuEB05NG+0T+wil0szK33345I4fx5jFbEaA+08++XQRLKOXbFSxD8k0ZCd/ws26hq1t JYujn3c7iCvPER4bZsvO0+BmHCkqm14bDWWlZMEnaFXZHqvlOm24VNEurNadcPpPaVl6 OKdwdnPMfJ6DDe7/bNi4OAfSC0NCpfAH1eKkULbg61L2m9o5KHpXWy3dn+ScAIMS/RX5 ru+g== X-Gm-Message-State: AJIora+/L9GD9tQa5RnYrPaGAticI0vQ3bbyMxVkEBtV3uBxizRvSvQi oQdLNgMjalcqaIuUwxlDmvF7e0aGKEYegg== X-Google-Smtp-Source: AGRyM1t7zJWBxWSNMaESaZzP4dL6iInO+BvFVUhQbAAwCutrOMehHM5eOGbROYM1AnmglXh7Xfnp1A== X-Received: by 2002:a17:902:e381:b0:16a:1b5d:5438 with SMTP id g1-20020a170902e38100b0016a1b5d5438mr26771585ple.147.1656048533524; Thu, 23 Jun 2022 22:28:53 -0700 (PDT) Received: from JMHNXMC7VH.bytedance.net ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id s144-20020a632c96000000b003c265b7d4f6sm602960pgs.44.2022.06.23.22.28.50 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 23 Jun 2022 22:28:53 -0700 (PDT) From: Han Xin To: hanxin.hx@bytedance.com Cc: chiyutianyi@gmail.com, derrickstolee@github.com, git@vger.kernel.org, haiyangtand@gmail.com, jonathantanmy@google.com, me@ttaylorr.com, ps@pks.im Subject: [PATCH v2 2/2] commit-graph.c: no lazy fetch in lookup_commit_in_graph() Date: Fri, 24 Jun 2022 13:27:57 +0800 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: <20220618030130.36419-1-hanxin.hx@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org If a commit is in the commit graph, we would expect the commit to also be present. So we should use has_object() instead of repo_has_object_file(), which will help us avoid getting into an endless loop of lazy fetch. When we found the commit in the graph in lookup_commit_in_graph(), but the commit is missing from the repository, we will try promisor_remote_get_direct() and then enter another loop. While sometimes it will finally succeed because it cannot fork subprocess, it has exhausted the local process resources and can be harmful to the remote service. Signed-off-by: Han Xin --- commit-graph.c | 2 +- t/t5329-no-lazy-fetch-with-commit-graph.sh | 47 ++++++++++++++++++++++ 2 files changed, 48 insertions(+), 1 deletion(-) create mode 100755 t/t5329-no-lazy-fetch-with-commit-graph.sh diff --git a/commit-graph.c b/commit-graph.c index 2b52818731..2dd9bcc7ea 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -907,7 +907,7 @@ struct commit *lookup_commit_in_graph(struct repository *repo, const struct obje return NULL; if (!search_commit_pos_in_graph(id, repo->objects->commit_graph, &pos)) return NULL; - if (!repo_has_object_file(repo, id)) + if (!has_object(repo, id, 0)) return NULL; commit = lookup_commit(repo, id); diff --git a/t/t5329-no-lazy-fetch-with-commit-graph.sh b/t/t5329-no-lazy-fetch-with-commit-graph.sh new file mode 100755 index 0000000000..4d25d2c950 --- /dev/null +++ b/t/t5329-no-lazy-fetch-with-commit-graph.sh @@ -0,0 +1,47 @@ +#!/bin/sh + +test_description='test for no lazy fetch with the commit-graph' + +. ./test-lib.sh + +test_expect_success 'setup: prepare a repository with a commit' ' + git init with-commit && + test_commit -C with-commit the-commit && + oid=$(git -C with-commit rev-parse HEAD) +' + +test_expect_success 'setup: prepare a repository with commit-graph contains the commit' ' + git init with-commit-graph && + echo "$(pwd)/with-commit/.git/objects" \ + >with-commit-graph/.git/objects/info/alternates && + # create a ref that points to the commit in alternates + git -C with-commit-graph update-ref refs/ref_to_the_commit "$oid" && + # prepare some other objects to commit-graph + test_commit -C with-commit-graph somthing && + git -c gc.writeCommitGraph=true -C with-commit-graph gc && + test_path_is_file with-commit-graph/.git/objects/info/commit-graph +' + +test_expect_success 'setup: change the alternates to what without the commit' ' + git init --bare without-commit && + echo "$(pwd)/without-commit/objects" \ + >with-commit-graph/.git/objects/info/alternates && + test_must_fail git -C with-commit-graph cat-file -e $oid +' + +test_expect_success 'setup: prepare another commit to fetch' ' + test_commit -C with-commit another-commit && + anycommit=$(git -C with-commit rev-parse HEAD) +' + +test_expect_success ULIMIT_PROCESSES 'fetch any commit from promisor with the usage of the commit graph' ' + git -C with-commit-graph remote add origin "$(pwd)/with-commit" && + git -C with-commit-graph config remote.origin.promisor true && + git -C with-commit-graph config remote.origin.partialclonefilter blob:none && + GIT_TRACE="$(pwd)/trace" run_with_limited_processses \ + git -C with-commit-graph fetch origin $anycommit 2>err && + test_i18ngrep ! "fatal: promisor-remote: unable to fork off fetch subprocess" err && + test $(grep "fetch origin" trace | wc -l) -eq 1 +' + +test_done