From patchwork Mon May 8 21:59:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1511C7EE22 for ; Mon, 8 May 2023 21:59:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234358AbjEHV75 (ORCPT ); Mon, 8 May 2023 17:59:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233109AbjEHV7q (ORCPT ); Mon, 8 May 2023 17:59:46 -0400 Received: from mail-yb1-xb2e.google.com (mail-yb1-xb2e.google.com [IPv6:2607:f8b0:4864:20::b2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 244BA114 for ; Mon, 8 May 2023 14:59:45 -0700 (PDT) Received: by mail-yb1-xb2e.google.com with SMTP id 3f1490d57ef6-b9a6f17f2b6so27434160276.1 for ; Mon, 08 May 2023 14:59:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583184; x=1686175184; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=MVGVfIy9Ojd38BYwc+JC4zC+NIIpENkuSEPRcWWSKuY=; b=RXXICgOgrxisOY4hSz8OIQf9Yzg+pk0SXuRRSleWmKEqmJKTa5u3/gY0Fw9fjl0Hof U4h3LPQtRmbJH4Dsd+1M/xA433y7FDhOVinGFWOyGY+Q9ABO8daaNShFXrFJkG8FzWXX xrZOBN+H6SrFyrZfoKFyS2d3ZqE0vfNnFup7GEgq0JS0zfxEOmS2mBgxQH5DAbOHgV8F YQH9D9GlVq8eJ1ACmIr5khl7an45nIS21h3kipozAwS2sCPul/ZDNFirL21yOjID7kfA +tlj/T5loItxBEIxc3mw5AaN8xjIYiR/L/H7pPtkyU9rD0HZC9wtoQBlywtslwQGwn6a dhgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583184; x=1686175184; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=MVGVfIy9Ojd38BYwc+JC4zC+NIIpENkuSEPRcWWSKuY=; b=BAQFuAx18nsZnADKoUY2GT5xTZ++3Q8H/pNJa1LCSRoD5Kt+Rr27WFDjjU7T+OMeEW EgSlRwWJyN6WUHZQm2nF8m9qbKAuitqGC89FKYHk9It/d9R0344L006Ag3VH+An3k68S NPwRXRSQJ/0OfQnyF3px9mZo5CkuIY3Uy/YtCUOO6+TxCyHzAAin3+O+3naihUhv2MNm q0+c0oqbV9fmsts6hYcgiovCDDfa0iDP5LJfQGiiqjhD41zNgL95XnPXXR2Kw84+00CK Hy4NkZbfQfhG5Xa6SDF8MeXw1MAvRT+FEXlofgNDfySQv7zuHI79p73hmNQUVM05bHxF KpnQ== X-Gm-Message-State: AC+VfDyIGuSyTOVKBOUMZfvJ46n32GVi41OKPXLHRordXWTheez1/HCh 1bmGScIJP6etIZNs6LseixJS1+msPVFUX5FenMWnxQ== X-Google-Smtp-Source: ACHHUZ5n0DNHOQTEugIEuAvcjmZLZuXb8oocO1HFGKMYBA/4JL8C+06BkGwK8Hico8tuGzL74Qrj8w== X-Received: by 2002:a81:12d8:0:b0:55a:89a2:e99c with SMTP id 207-20020a8112d8000000b0055a89a2e99cmr13666612yws.3.1683583184210; Mon, 08 May 2023 14:59:44 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id n1-20020a0dcb01000000b0055a503ca1e8sm2777243ywd.109.2023.05.08.14.59.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 14:59:43 -0700 (PDT) Date: Mon, 8 May 2023 17:59:42 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 01/15] refs.c: rename `ref_filter` Message-ID: <2225f799410687ae72f6f64daaf93809e7c9b380.1683581621.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King The refs machinery has its own implementation of a `ref_filter` (used by `for-each-ref`), which is distinct from the `ref-filler.h` API (also used by `for-each-ref`, among other things). Rename the one within refs.c to more clearly indicate its purpose. Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- refs.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/refs.c b/refs.c index d2a98e1c21..b9b77d2eff 100644 --- a/refs.c +++ b/refs.c @@ -375,8 +375,8 @@ char *resolve_refdup(const char *refname, int resolve_flags, oid, flags); } -/* The argument to filter_refs */ -struct ref_filter { +/* The argument to for_each_filter_refs */ +struct for_each_ref_filter { const char *pattern; const char *prefix; each_ref_fn *fn; @@ -409,10 +409,11 @@ int ref_exists(const char *refname) return refs_ref_exists(get_main_ref_store(the_repository), refname); } -static int filter_refs(const char *refname, const struct object_id *oid, - int flags, void *data) +static int for_each_filter_refs(const char *refname, + const struct object_id *oid, + int flags, void *data) { - struct ref_filter *filter = (struct ref_filter *)data; + struct for_each_ref_filter *filter = data; if (wildmatch(filter->pattern, refname, 0)) return 0; @@ -569,7 +570,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern, const char *prefix, void *cb_data) { struct strbuf real_pattern = STRBUF_INIT; - struct ref_filter filter; + struct for_each_ref_filter filter; int ret; if (!prefix && !starts_with(pattern, "refs/")) @@ -589,7 +590,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern, filter.prefix = prefix; filter.fn = fn; filter.cb_data = cb_data; - ret = for_each_ref(filter_refs, &filter); + ret = for_each_ref(for_each_filter_refs, &filter); strbuf_release(&real_pattern); return ret; From patchwork Mon May 8 21:59:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF50EC7EE25 for ; Mon, 8 May 2023 22:00:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234383AbjEHV77 (ORCPT ); Mon, 8 May 2023 17:59:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234386AbjEHV7v (ORCPT ); Mon, 8 May 2023 17:59:51 -0400 Received: from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com [IPv6:2607:f8b0:4864:20::1134]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D4174EFC for ; Mon, 8 May 2023 14:59:48 -0700 (PDT) Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-559e317eef1so75350947b3.0 for ; Mon, 08 May 2023 14:59:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583187; x=1686175187; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=vdKIcqYBgzsFlc3rxsHPqqlUt5NsrdUU8Q1NvrDtse0=; b=IRMH5LvRrPaTrjNDbQ3IgArpKtJASdkvZvbiFMvMGuMSsE6h5aq9B7ZY7T9ftGQ4Tm 4jc6NcgsKUDH1XRffpXmpf1gn9gT2a8kijBt0A7PtQQP8XEdjHYzCim4QsnrxoGemMyn kkMnYvqi+M+8gCOoAdvzCaewMtaReRIsYaA4dGQfSIA3rdzFLoX8/YFxp51JDAUX9+Pr DNwRrchJPcxZodjB8R9xMBmwKZtKNfpPXMgxexT7YkcRqor2fqlweT4+f5wTl+43uHS3 f/8xZfzpOKSXZyTbUR6BGipbNTEY6dnrQBv/4ix3KG3gcKbDgF24W7XPNAsjKSv9tlzp A0Gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583187; x=1686175187; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=vdKIcqYBgzsFlc3rxsHPqqlUt5NsrdUU8Q1NvrDtse0=; b=OBdNBruWLU7skitDaad8CqKbuoRdi7Ri2iwyqhhLv+XoHX1oM2FVfLdH5X8pi0VHnl AJTPBA9epbc7TbtOyWebZ9C1XuuA23cVmamkxX1cvtAhXMr132iYT9X4YdBvLighrjh8 k0kbV6JtBxMB2Gu2cVAgKZrNscaenBvIwUDFdZZxBjwycmtlWk5T+8V60CZwSv3stV+D j9hcM1Y5ZO20wx5GnvZz0yzmssGNQQrZOD0nxY2vMOHL4rvwFODghBjwgLKCgQDeQKdp 3wRq/+1sT1sMJIs3aj/a96ORwSOhtrCaHV+0r8a8zwWCRV/+IQ2io8nhAH4nvySUUs+Q aPzQ== X-Gm-Message-State: AC+VfDzAT762pSJAu6m8BZ2bCDwfm8tg+jf2WizH5JzLx/KvKDmseD71 29/jCQ52xiljMB37EiwmL8fF2t1pUGYaVpI/EN/8dg== X-Google-Smtp-Source: ACHHUZ5+ynZDlkQeCId48RgpcNlpKe75pJ7dBgv5rlDQ07pEBZ3yaOSdulQ3bkk7wumMe7cPDHJnZQ== X-Received: by 2002:a0d:f1c3:0:b0:55d:aff9:fbbb with SMTP id a186-20020a0df1c3000000b0055daff9fbbbmr14133605ywf.28.1683583187222; Mon, 08 May 2023 14:59:47 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id u124-20020a816082000000b0055a1c6d1249sm2791667ywb.72.2023.05.08.14.59.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 14:59:46 -0700 (PDT) Date: Mon, 8 May 2023 17:59:45 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 02/15] ref-filter.h: provide `REF_FILTER_INIT` Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King Provide a sane initialization value for `struct ref_filter`, which in a subsequent patch will be used to initialize a new field. In the meantime, fix a case in test-reach.c where its `ref_filter` is not even zero-initialized. Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- builtin/branch.c | 3 +-- builtin/for-each-ref.c | 3 +-- builtin/tag.c | 3 +-- ref-filter.h | 3 +++ t/helper/test-reach.c | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/builtin/branch.c b/builtin/branch.c index 501c47657c..03bb8e414c 100644 --- a/builtin/branch.c +++ b/builtin/branch.c @@ -662,7 +662,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix) int reflog = 0, quiet = 0, icase = 0, force = 0, recurse_submodules_explicit = 0; enum branch_track track; - struct ref_filter filter; + struct ref_filter filter = REF_FILTER_INIT; static struct ref_sorting *sorting; struct string_list sorting_options = STRING_LIST_INIT_DUP; struct ref_format format = REF_FORMAT_INIT; @@ -720,7 +720,6 @@ int cmd_branch(int argc, const char **argv, const char *prefix) setup_ref_filter_porcelain_msg(); - memset(&filter, 0, sizeof(filter)); filter.kind = FILTER_REFS_BRANCHES; filter.abbrev = -1; diff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c index 695fc8f4a5..99ccb73518 100644 --- a/builtin/for-each-ref.c +++ b/builtin/for-each-ref.c @@ -24,7 +24,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix) struct string_list sorting_options = STRING_LIST_INIT_DUP; int maxcount = 0, icase = 0, omit_empty = 0; struct ref_array array; - struct ref_filter filter; + struct ref_filter filter = REF_FILTER_INIT; struct ref_format format = REF_FORMAT_INIT; struct strbuf output = STRBUF_INIT; struct strbuf err = STRBUF_INIT; @@ -61,7 +61,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix) }; memset(&array, 0, sizeof(array)); - memset(&filter, 0, sizeof(filter)); format.format = "%(objectname) %(objecttype)\t%(refname)"; diff --git a/builtin/tag.c b/builtin/tag.c index 1850a6a6fd..6b41bb7374 100644 --- a/builtin/tag.c +++ b/builtin/tag.c @@ -443,7 +443,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix) struct msg_arg msg = { .buf = STRBUF_INIT }; struct ref_transaction *transaction; struct strbuf err = STRBUF_INIT; - struct ref_filter filter; + struct ref_filter filter = REF_FILTER_INIT; struct ref_sorting *sorting; struct string_list sorting_options = STRING_LIST_INIT_DUP; struct ref_format format = REF_FORMAT_INIT; @@ -501,7 +501,6 @@ int cmd_tag(int argc, const char **argv, const char *prefix) git_config(git_tag_config, &sorting_options); memset(&opt, 0, sizeof(opt)); - memset(&filter, 0, sizeof(filter)); filter.lines = -1; opt.sign = -1; diff --git a/ref-filter.h b/ref-filter.h index 430701cfb7..a920f73b29 100644 --- a/ref-filter.h +++ b/ref-filter.h @@ -92,6 +92,9 @@ struct ref_format { struct string_list bases; }; +#define REF_FILTER_INIT { \ + .points_at = OID_ARRAY_INIT, \ +} #define REF_FORMAT_INIT { \ .use_color = -1, \ .bases = STRING_LIST_INIT_DUP, \ diff --git a/t/helper/test-reach.c b/t/helper/test-reach.c index 5b6f217441..ef58f10c2d 100644 --- a/t/helper/test-reach.c +++ b/t/helper/test-reach.c @@ -139,7 +139,7 @@ int cmd__reach(int ac, const char **av) printf("%s(X,_,_,0,0):%d\n", av[1], can_all_from_reach_with_flag(&X_obj, 2, 4, 0, 0)); } else if (!strcmp(av[1], "commit_contains")) { - struct ref_filter filter; + struct ref_filter filter = REF_FILTER_INIT; struct contains_cache cache; init_contains_cache(&cache); From patchwork Mon May 8 21:59:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86FDEC77B7F for ; Mon, 8 May 2023 21:59:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234345AbjEHV76 (ORCPT ); Mon, 8 May 2023 17:59:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39498 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234413AbjEHV7x (ORCPT ); Mon, 8 May 2023 17:59:53 -0400 Received: from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com [IPv6:2607:f8b0:4864:20::1134]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E3DA5241 for ; Mon, 8 May 2023 14:59:51 -0700 (PDT) Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-55a83e80262so75974147b3.3 for ; Mon, 08 May 2023 14:59:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583190; x=1686175190; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=OmOhmOrxJpJeyICgzWAKI9cAhqjkipbsCj5HP5Ilgfk=; b=5p33H01NHscU9zB3BXW+UdRr1Z9udRlBYs9CmmoHw1+Fucbu4hNYrqjF0/KlLtTq1I A2dYfZ9zQInIHlGweDoHT9gjDJgS3vVhNtvYL/PxD50ptE5X9SSqNU6TwvXlXdNU/bUV aSsu3opJ5W6b/+Ed6PRYOdT0ZjibOBBCzSbqC00TRnOkwwPrnMKd/yHoYbVR4Tc7nAS1 CVDIZO0INclN2jmE6vuX+kKByphAOx8BUSZMfa2qs65njqYmvH5huKZdo9P2uxQnxMrn SgySpvEdYTAKSSDg0EIV4TcbLfy4MzDo7kWLhet2QILP/8h3qvvs1N/9/9FhzlGC03JC 4cMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583190; x=1686175190; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=OmOhmOrxJpJeyICgzWAKI9cAhqjkipbsCj5HP5Ilgfk=; b=fkiE7bQyOQTKE7PjTAP02J9S09FQVPtv+p4h/9gx6I5iQYIgEG0VQZJWnO81MpFik7 i5k2l/tVlXEvwmUfF/gPzdkPRw6ZOUSUZXBF0W7/IfnG55xF7nYlXbnpnZVfPkd06A5M GcLVbS2bbNcqBaxol/bBn3ZBvZKrcg8xxc7XHNLqI/9wnNrRJyaV0nSRiKRY0PirFDfq 4pZracUCFA255mJoX88sZwEfsOMidvOFpZfyACDKYRtLMmtd01I8TWYKbX7h8BqiVPmP bwCbLidlhUjwsU+NAwviKEaJtujtLZusXlHYAmLgXaAzR3UzGNYtO2mRKTrEtiSgq6Rs FvsA== X-Gm-Message-State: AC+VfDwXb7GUFecqnKK396439vwQoeoas0sGXhjFBZgInF8ic5t1NcT2 Dfhagb0bXOnG5Y2ZPlS6RnWLL1OfcNbO26tcc3jg7g== X-Google-Smtp-Source: ACHHUZ77yGlalaG9MzSc7eQpUTRnKzHbhf5oHzJNJHqszDDcxkUEVOqhjUJjoLX8JbpT0dT55l4BlA== X-Received: by 2002:a81:5e41:0:b0:55a:985f:848d with SMTP id s62-20020a815e41000000b0055a985f848dmr14643481ywb.39.1683583190414; Mon, 08 May 2023 14:59:50 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id t204-20020a0dead5000000b0055a1069886fsm2783339ywe.129.2023.05.08.14.59.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 14:59:49 -0700 (PDT) Date: Mon, 8 May 2023 17:59:48 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 03/15] ref-filter: clear reachable list pointers after freeing Message-ID: <7fe8623f606d7abf20b39b24a849d4f8aad6ecec.1683581621.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King In reach_filter(), we pop all commits from the reachable lists, leaving them empty. But because we're operating on a list pointer that was passed by value, the original filter.reachable_from pointer is left dangling. Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- ref-filter.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/ref-filter.c b/ref-filter.c index 10aab14f03..b1d5022a51 100644 --- a/ref-filter.c +++ b/ref-filter.c @@ -2416,13 +2416,13 @@ void ref_array_clear(struct ref_array *array) #define EXCLUDE_REACHED 0 #define INCLUDE_REACHED 1 static void reach_filter(struct ref_array *array, - struct commit_list *check_reachable, + struct commit_list **check_reachable, int include_reached) { int i, old_nr; struct commit **to_clear; - if (!check_reachable) + if (!*check_reachable) return; CALLOC_ARRAY(to_clear, array->nr); @@ -2432,7 +2432,7 @@ static void reach_filter(struct ref_array *array, } tips_reachable_from_bases(the_repository, - check_reachable, + *check_reachable, to_clear, array->nr, UNINTERESTING); @@ -2453,8 +2453,8 @@ static void reach_filter(struct ref_array *array, clear_commit_marks_many(old_nr, to_clear, ALL_REV_FLAGS); - while (check_reachable) { - struct commit *merge_commit = pop_commit(&check_reachable); + while (*check_reachable) { + struct commit *merge_commit = pop_commit(check_reachable); clear_commit_marks(merge_commit, ALL_REV_FLAGS); } @@ -2551,8 +2551,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int clear_contains_cache(&ref_cbdata.no_contains_cache); /* Filters that need revision walking */ - reach_filter(array, filter->reachable_from, INCLUDE_REACHED); - reach_filter(array, filter->unreachable_from, EXCLUDE_REACHED); + reach_filter(array, &filter->reachable_from, INCLUDE_REACHED); + reach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED); save_commit_buffer = save_commit_buffer_orig; return ret; From patchwork Mon May 8 21:59:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3618AC7EE26 for ; Mon, 8 May 2023 22:00:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234446AbjEHWAB (ORCPT ); Mon, 8 May 2023 18:00:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234439AbjEHV7z (ORCPT ); Mon, 8 May 2023 17:59:55 -0400 Received: from mail-yw1-x1129.google.com (mail-yw1-x1129.google.com [IPv6:2607:f8b0:4864:20::1129]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81E8CE70 for ; Mon, 8 May 2023 14:59:54 -0700 (PDT) Received: by mail-yw1-x1129.google.com with SMTP id 00721157ae682-55a5a830238so46493247b3.3 for ; Mon, 08 May 2023 14:59:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583193; x=1686175193; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=VDtx3iqE2RZQuNCLKAySEpkN7cP3UMLjgoQpHks3AJY=; b=FaJpuR6pZNfm/ACv2/OO35hYdztb22m0HTC9LW+XQlvx2JLWYlziVJdRnyZ1YCmry+ SMNVTcmfLPK0KgZYTPGPswDxx8Jbu4+paXkDAUmlp3QahbF+nHiI13vlbq1NQjvan0BG oahslJ+TkX5yoy3jGat8MJFkpAkgWDLpqBW+dUee3xt0hQ8xdlUao/a0zk81ra4npBV+ 4kPtHHx9AtvvdAx8fGN3UvdXtAIoR37KOldilg7Aok56lTzpk6GfUeTIGPjTjHL6xnTh aVNvMZo19S6aYFaB/xchVCsmS1KmCmBF0PYcJYTy85yqO0QvzfzppPdcEKaD4oBqbVFQ zMFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583193; x=1686175193; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=VDtx3iqE2RZQuNCLKAySEpkN7cP3UMLjgoQpHks3AJY=; b=G4Nkzt3+Qdje1e0nqA9jKpo44+AOMQ9IhnZPGFhw7h0xj+OiamfcAHL+UHh9I+Eysx +Hf7PidRsPnnZOEs2DTV4+7Ol5J3SHiGYZIKcBs2DIRP4x2rT4iaSHhUbHNY32HNjyQ6 VC6UnS8dH3qllt/14uvUaVlRgpDUE1VeN9sMdnsKY3aTZOUPHQRXrdG3cupkxXlsPakE fPPRgjQVSv6J/YOOUClXarda5j7y9K+OIaBLJNJ2g22VjEzF5zBEP9d/TuMu8kykLOM/ izH/6Yjrxf/xMOD+cehtNytzbxKZ+RvJnKarzozTbjEiRjTg0RcI6Db2oqxqK3ybGAgV YFHw== X-Gm-Message-State: AC+VfDzRN0A9SfVBZEV1aG8Wb12VxNbE7XWuG+7MuagyfuaAAYKudNrg yNdGRRyLSj6PovOqv6QL+Yjf3tcOhMj1upH9sniePQ== X-Google-Smtp-Source: ACHHUZ43C7ZvBQGJYvLJeJ8V2MtRD/fH34TJ4EIQeNd0cz9wweKzhn6GgNFwEPWQPY8KSiH/QHq9CQ== X-Received: by 2002:a81:5404:0:b0:55d:c333:26c4 with SMTP id i4-20020a815404000000b0055dc33326c4mr8236520ywb.0.1683583193387; Mon, 08 May 2023 14:59:53 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id g142-20020a815294000000b0055d6b67f468sm2795753ywb.14.2023.05.08.14.59.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 14:59:53 -0700 (PDT) Date: Mon, 8 May 2023 17:59:52 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 04/15] ref-filter: add ref_filter_clear() Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King We did not bother to clean up at all in branch/tag, and for-each-ref only hit a few elements. So this is probably cleaning up leaks, but I didn't check yet. Note that the reachable_from and unreachable_from lists should be cleaned as they are used. So this is just covering any case where we might bail before running the reachability check. --- builtin/branch.c | 1 + builtin/for-each-ref.c | 3 +-- builtin/tag.c | 1 + ref-filter.c | 16 ++++++++++++++++ ref-filter.h | 3 +++ 5 files changed, 22 insertions(+), 2 deletions(-) diff --git a/builtin/branch.c b/builtin/branch.c index 03bb8e414c..c201f0cb0b 100644 --- a/builtin/branch.c +++ b/builtin/branch.c @@ -813,6 +813,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix) print_columns(&output, colopts, NULL); string_list_clear(&output, 0); ref_sorting_release(sorting); + ref_filter_clear(&filter); return 0; } else if (edit_description) { const char *branch_name; diff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c index 99ccb73518..c01fa6fefe 100644 --- a/builtin/for-each-ref.c +++ b/builtin/for-each-ref.c @@ -120,8 +120,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix) strbuf_release(&err); strbuf_release(&output); ref_array_clear(&array); - free_commit_list(filter.with_commit); - free_commit_list(filter.no_commit); + ref_filter_clear(&filter); ref_sorting_release(sorting); strvec_clear(&vec); return 0; diff --git a/builtin/tag.c b/builtin/tag.c index 6b41bb7374..aab5e693fe 100644 --- a/builtin/tag.c +++ b/builtin/tag.c @@ -645,6 +645,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix) cleanup: ref_sorting_release(sorting); + ref_filter_clear(&filter); strbuf_release(&buf); strbuf_release(&ref); strbuf_release(&reflog_msg); diff --git a/ref-filter.c b/ref-filter.c index b1d5022a51..9ea92b9637 100644 --- a/ref-filter.c +++ b/ref-filter.c @@ -2864,3 +2864,19 @@ int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset) return 0; } + +void ref_filter_init(struct ref_filter *filter) +{ + struct ref_filter blank = REF_FILTER_INIT; + memcpy(filter, &blank, sizeof(blank)); +} + +void ref_filter_clear(struct ref_filter *filter) +{ + oid_array_clear(&filter->points_at); + free_commit_list(filter->with_commit); + free_commit_list(filter->no_commit); + free_commit_list(filter->reachable_from); + free_commit_list(filter->unreachable_from); + ref_filter_init(filter); +} diff --git a/ref-filter.h b/ref-filter.h index a920f73b29..160b807224 100644 --- a/ref-filter.h +++ b/ref-filter.h @@ -170,4 +170,7 @@ void filter_ahead_behind(struct repository *r, struct ref_format *format, struct ref_array *array); +void ref_filter_init(struct ref_filter *filter); +void ref_filter_clear(struct ref_filter *filter); + #endif /* REF_FILTER_H */ From patchwork Mon May 8 21:59:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F73CC77B7F for ; Mon, 8 May 2023 22:00:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234401AbjEHWAC (ORCPT ); Mon, 8 May 2023 18:00:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234346AbjEHV76 (ORCPT ); Mon, 8 May 2023 17:59:58 -0400 Received: from mail-yb1-xb2f.google.com (mail-yb1-xb2f.google.com [IPv6:2607:f8b0:4864:20::b2f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD2A7102 for ; Mon, 8 May 2023 14:59:57 -0700 (PDT) Received: by mail-yb1-xb2f.google.com with SMTP id 3f1490d57ef6-b9e66ce80acso6893176276.3 for ; Mon, 08 May 2023 14:59:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583197; x=1686175197; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=OZd3f4LH9FIps4H7kT3AlcP2vjt1ARQg19e8+HO6wZ4=; b=k7m6VTBwTr8rAf8zwDG5KA+5iFOj4DQK0lQ9rpvF8DLKNbfkbEHBvsdjBxreM2XRL3 hhZXEhgF3J995KdDQXoK1FurYiteGT/XbtjUS6UmJKrF7FjI8jZbFmDF6UpyOfTo/B51 eFUaBRNgCttkEPLUWTpbCrCszTA+F7K4kWOs/wrHXkG7Dtq2svH9OJ9R03I5wGileLzg ZnyoySyzb68d4kZ9ilo8dWDT7WYb+tpoZl/idcmwKeabMFRp8DNM/Ko98asptRrJ4bsZ hcsM9bn93zFvv0puDViYmPijB/1o4FXffHz9VQMLHbQrg7uu1XuPKAmSZ58+HxLW6Hv1 4cRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583197; x=1686175197; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=OZd3f4LH9FIps4H7kT3AlcP2vjt1ARQg19e8+HO6wZ4=; b=bEwfI5cDi26ifFpUH22ycSnmLsHc46mONbKd+W/+nSQdvFYHj100q7vkxJwZJogvu6 wjfKJIH3m3Au17Op2AdJxJ0iaoIzIgk+FAKzIbyru9s8u+8RmwTeQaZH/0e36brLNf/a N5eBWL7I9HP3Gm5Zskxos1087WNF70jVHiE5cr3aDn8hpe+RFbn35LEzR3IMNnaI7Bss YOYFLTHH82g9EpHS2/ww7B8FtWoZfdKusZmgHIMQXyx0giJ7MdpDtPuMlIAWpBBp5Jkf 8eMA6HNBsnBVeSgiUdk1TeNWbpur29WToNbs7NM7n1W+tvrRnI9WaSTE9I0YU3cOuUfz /05Q== X-Gm-Message-State: AC+VfDwJ6WwZ0OXwAziD6B7FYgzW12aLvx3yEQLSUywcsl+kBN/r6Aua z/A/5k2UGIxscCssXw1ILx9ZBbPbk80o1qazdGwatw== X-Google-Smtp-Source: ACHHUZ4LRssufuqPKR+Gl8yV9qGIxu4TgJUzCuw2ny2omb19g2IgVx/5tFGAMPwOaARxzIeS8yekfQ== X-Received: by 2002:a25:addf:0:b0:b8f:599c:c43f with SMTP id d31-20020a25addf000000b00b8f599cc43fmr14077836ybe.19.1683583196860; Mon, 08 May 2023 14:59:56 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id k3-20020a25ed03000000b00b8f4916294csm2577988ybh.10.2023.05.08.14.59.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 14:59:56 -0700 (PDT) Date: Mon, 8 May 2023 17:59:55 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 05/15] ref-filter.c: parameterize match functions over patterns Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King `match_pattern()` and `match_name_as_path()` both take a `struct ref_filter *`, and then store a stack variable `patterns` pointing at `filter->patterns`. The subsequent patch will add a new array of patterns to match over (the excluded patterns, via a new `git for-each-ref --exclude` option), treating the return value of these functions differently depending on which patterns are being used to match. Tweak `match_pattern()` and `match_name_as_path()` to take an array of patterns to prepare for passing either in. Co-authored-by: Taylor Blau Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- ref-filter.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/ref-filter.c b/ref-filter.c index 9ea92b9637..6c5eed144f 100644 --- a/ref-filter.c +++ b/ref-filter.c @@ -2102,9 +2102,10 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom, * matches a pattern "refs/heads/mas") or a wildcard (e.g. the same ref * matches "refs/heads/mas*", too). */ -static int match_pattern(const struct ref_filter *filter, const char *refname) +static int match_pattern(const struct ref_filter *filter, + const char **patterns, + const char *refname) { - const char **patterns = filter->name_patterns; unsigned flags = 0; if (filter->ignore_case) @@ -2132,9 +2133,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname) * matches a pattern "refs/heads/" but not "refs/heads/m") or a * wildcard (e.g. the same ref matches "refs/heads/m*", too). */ -static int match_name_as_path(const struct ref_filter *filter, const char *refname) +static int match_name_as_path(const struct ref_filter *filter, + const char **pattern, + const char *refname) { - const char **pattern = filter->name_patterns; int namelen = strlen(refname); unsigned flags = WM_PATHNAME; @@ -2163,8 +2165,8 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname) if (!*filter->name_patterns) return 1; /* No pattern always matches */ if (filter->match_as_path) - return match_name_as_path(filter, refname); - return match_pattern(filter, refname); + return match_name_as_path(filter, filter->name_patterns, refname); + return match_pattern(filter, filter->name_patterns, refname); } /* From patchwork Mon May 8 21:59:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF39DC77B7F for ; Mon, 8 May 2023 22:00:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234459AbjEHWAR (ORCPT ); Mon, 8 May 2023 18:00:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234471AbjEHWAL (ORCPT ); Mon, 8 May 2023 18:00:11 -0400 Received: from mail-yw1-x112c.google.com (mail-yw1-x112c.google.com [IPv6:2607:f8b0:4864:20::112c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3F3A47EFA for ; Mon, 8 May 2023 15:00:01 -0700 (PDT) Received: by mail-yw1-x112c.google.com with SMTP id 00721157ae682-55a76ed088aso76171997b3.2 for ; Mon, 08 May 2023 15:00:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583200; x=1686175200; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=/wB4zu9ImKbyCdyVEKtkbU9Q3qjrKZMrVMOOvObJe/Q=; b=QSwEPjwHyEq95pFBVcIATrvyo/oKHiw/YHlz00G1tKerB865jfxKffw25URsyUD1fT ye4s/ivRukASuSw2KBsrRBbuYGDrrQRlVleIidqObxaqflZYqQf1pk0VzDRgrKXQ7R2R wQ/vF13xa2S5a0Kg77Te/1Nh8yupmlyeuZKGy8B1TELDApq5UNK1Y5uKxZa/aPsIxY/A lxXeerbHtjiqiPmHjF8mRWskYk0js4GfKSl8rAi0rFIum1XIg9Tq0qTh4fr0AB/T6Ct5 45xwlOI7uK+puXUHBC+e9sQ/FlrypcNDIBYjYmIVf8gwweqUtQo5bimyaA6j8/HLQ9Pf TR8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583200; x=1686175200; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/wB4zu9ImKbyCdyVEKtkbU9Q3qjrKZMrVMOOvObJe/Q=; b=jiddvTeLNc7oR6YIRP2vOy7z5f077T8KfhkyvdQnlPxpwxnOzpMaPNqcF61J9jCa/p 4R7eiHFYNs1bLhZyDiWoXlt9iBC9kQPdL2Pwcp6/OZonrQ586luxUz7t9A2XYj97k4v+ croraFsoOQgf2xHsb3Al9I0SKXLhABNONlpna1mgGJ0VvrjDCFCYZjMo6MLHyvudyYta 2U2PdM2CWUjbJBMo1GG6H7cxWCpO/qY3gMwknRZw7vBhPwf3aCMV+xrpJ93wbAurCFop FrbzIchZggNaLuv3yM+eS3Knf/MymR42u13dKp8jcnlGPtxONy5YogykcBjs+lwsYHZP w5VA== X-Gm-Message-State: AC+VfDxrIjsG4rNGy2UbB2hdXLpkhb7yfOq8qpEfzbLZXiON3o/n6GpL jDfRZbUmhIF+CUJOlqFCEjgLZOcAoYQmRF+kaYz7QA== X-Google-Smtp-Source: ACHHUZ5HzcgkeFUybjk1x3cCe9VjVmY25KhpKDcBRfnRITuLqtBNheVUhk3O9Y5e/M1DFFyoKiVXdA== X-Received: by 2002:a0d:db49:0:b0:55a:314d:afdc with SMTP id d70-20020a0ddb49000000b0055a314dafdcmr13346687ywe.37.1683583200054; Mon, 08 May 2023 15:00:00 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id n1-20020a0dcb01000000b0055a503ca1e8sm2777385ywd.109.2023.05.08.14.59.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 14:59:59 -0700 (PDT) Date: Mon, 8 May 2023 17:59:58 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 06/15] builtin/for-each-ref.c: add `--exclude` option Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When using `for-each-ref`, it is sometimes convenient for the caller to be able to exclude certain parts of the references. For example, if there are many `refs/__hidden__/*` references, the caller may want to emit all references *except* the hidden ones. Currently, the only way to do this is to post-process the output, like: $ git for-each-ref --format='%(refname)' | grep -v '^refs/hidden/' Which is do-able, but requires processing a potentially large quantity of references. Teach `git for-each-ref` a new `--exclude=` option, which excludes references from the results if they match one or more excluded patterns. This patch provides a naive implementation where the `ref_filter` still sees all references (including ones that it will discard) and is left to check whether each reference matches any excluded pattern(s) before emitting them. By culling out references we know the caller doesn't care about, we can avoid allocating memory for their storage, as well as spending time sorting the output (among other things). Even the naive implementation provides a significant speed-up on a modified copy of linux.git (that has a hidden ref pointing at each commit): $ hyperfine \ 'git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"' \ 'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/' Benchmark 1: git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/" Time (mean ± σ): 820.1 ms ± 2.0 ms [User: 703.7 ms, System: 152.0 ms] Range (min … max): 817.7 ms … 823.3 ms 10 runs Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/ Time (mean ± σ): 106.6 ms ± 1.1 ms [User: 99.4 ms, System: 7.1 ms] Range (min … max): 104.7 ms … 109.1 ms 27 runs Summary 'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/' ran 7.69 ± 0.08 times faster than 'git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"' Subsequent patches will improve on this by avoiding visiting excluded sections of the `packed-refs` file in certain cases. Co-authored-by: Jeff King Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- Documentation/git-for-each-ref.txt | 6 +++++ builtin/for-each-ref.c | 2 ++ ref-filter.c | 13 +++++++++++ ref-filter.h | 6 +++++ t/t6300-for-each-ref.sh | 35 ++++++++++++++++++++++++++++++ 5 files changed, 62 insertions(+) diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt index 1e215d4e73..5743eb5def 100644 --- a/Documentation/git-for-each-ref.txt +++ b/Documentation/git-for-each-ref.txt @@ -14,6 +14,7 @@ SYNOPSIS [--points-at=] [--merged[=]] [--no-merged[=]] [--contains[=]] [--no-contains[=]] + [--exclude= ...] DESCRIPTION ----------- @@ -102,6 +103,11 @@ OPTIONS Do not print a newline after formatted refs where the format expands to the empty string. +--exclude=:: + If one or more patterns are given, only refs which do not match + any excluded pattern(s) are shown. Matching is done using the + same rules as `` above. + FIELD NAMES ----------- diff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c index c01fa6fefe..449da61e11 100644 --- a/builtin/for-each-ref.c +++ b/builtin/for-each-ref.c @@ -14,6 +14,7 @@ static char const * const for_each_ref_usage[] = { N_("git for-each-ref [--points-at ]"), N_("git for-each-ref [--merged []] [--no-merged []]"), N_("git for-each-ref [--contains []] [--no-contains []]"), + N_("git for-each-ref [--exclude= ...]"), NULL }; @@ -47,6 +48,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix) OPT_INTEGER( 0 , "count", &maxcount, N_("show only matched refs")), OPT_STRING( 0 , "format", &format.format, N_("format"), N_("format to use for the output")), OPT__COLOR(&format.use_color, N_("respect format colors")), + OPT_REF_FILTER_EXCLUDE(&filter), OPT_REF_SORT(&sorting_options), OPT_CALLBACK(0, "points-at", &filter.points_at, N_("object"), N_("print only refs which points at the given object"), diff --git a/ref-filter.c b/ref-filter.c index 6c5eed144f..93dc9b331f 100644 --- a/ref-filter.c +++ b/ref-filter.c @@ -2169,6 +2169,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname) return match_pattern(filter, filter->name_patterns, refname); } +static int filter_exclude_match(struct ref_filter *filter, const char *refname) +{ + if (!filter->exclude.nr) + return 0; + if (filter->match_as_path) + return match_name_as_path(filter, filter->exclude.v, refname); + return match_pattern(filter, filter->exclude.v, refname); +} + /* * This is the same as for_each_fullref_in(), but it tries to iterate * only over the patterns we'll care about. Note that it _doesn't_ do a full @@ -2336,6 +2345,9 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid, if (!filter_pattern_match(filter, refname)) return 0; + if (filter_exclude_match(filter, refname)) + return 0; + if (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname)) return 0; @@ -2875,6 +2887,7 @@ void ref_filter_init(struct ref_filter *filter) void ref_filter_clear(struct ref_filter *filter) { + strvec_clear(&filter->exclude); oid_array_clear(&filter->points_at); free_commit_list(filter->with_commit); free_commit_list(filter->no_commit); diff --git a/ref-filter.h b/ref-filter.h index 160b807224..1524bc463a 100644 --- a/ref-filter.h +++ b/ref-filter.h @@ -6,6 +6,7 @@ #include "refs.h" #include "commit.h" #include "string-list.h" +#include "strvec.h" /* Quoting styles */ #define QUOTE_NONE 0 @@ -59,6 +60,7 @@ struct ref_array { struct ref_filter { const char **name_patterns; + struct strvec exclude; struct oid_array points_at; struct commit_list *with_commit; struct commit_list *no_commit; @@ -94,6 +96,7 @@ struct ref_format { #define REF_FILTER_INIT { \ .points_at = OID_ARRAY_INIT, \ + .exclude = STRVEC_INIT, \ } #define REF_FORMAT_INIT { \ .use_color = -1, \ @@ -112,6 +115,9 @@ struct ref_format { #define OPT_REF_SORT(var) \ OPT_STRING_LIST(0, "sort", (var), \ N_("key"), N_("field name to sort on")) +#define OPT_REF_FILTER_EXCLUDE(var) \ + OPT_STRVEC(0, "exclude", &(var)->exclude, \ + N_("pattern"), N_("exclude refs which match pattern")) /* * API for filtering a set of refs. Based on the type of refs the user diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh index 5c00607608..7e8d578522 100755 --- a/t/t6300-for-each-ref.sh +++ b/t/t6300-for-each-ref.sh @@ -447,6 +447,41 @@ test_expect_success 'exercise glob patterns with prefixes' ' test_cmp expected actual ' +cat >expected <<\EOF +refs/tags/bar +refs/tags/baz +refs/tags/testtag +EOF + +test_expect_success 'exercise patterns with prefix exclusions' ' + for tag in foo/one foo/two foo/three bar baz + do + git tag "$tag" || return 1 + done && + test_when_finished "git tag -d foo/one foo/two foo/three bar baz" && + git for-each-ref --format="%(refname)" \ + refs/tags/ --exclude=refs/tags/foo >actual && + test_cmp expected actual +' + +cat >expected <<\EOF +refs/tags/bar +refs/tags/baz +refs/tags/foo/one +refs/tags/testtag +EOF + +test_expect_success 'exercise patterns with pattern exclusions' ' + for tag in foo/one foo/two foo/three bar baz + do + git tag "$tag" || return 1 + done && + test_when_finished "git tag -d foo/one foo/two foo/three bar baz" && + git for-each-ref --format="%(refname)" \ + refs/tags/ --exclude="refs/tags/foo/t*" >actual && + test_cmp expected actual +' + cat >expected <<\EOF 'refs/heads/main' 'refs/remotes/origin/main' From patchwork Mon May 8 22:00:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9846DC7EE22 for ; Mon, 8 May 2023 22:00:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234460AbjEHWAY (ORCPT ); Mon, 8 May 2023 18:00:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234461AbjEHWAN (ORCPT ); Mon, 8 May 2023 18:00:13 -0400 Received: from mail-yb1-xb2d.google.com (mail-yb1-xb2d.google.com [IPv6:2607:f8b0:4864:20::b2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 789A083E3 for ; Mon, 8 May 2023 15:00:04 -0700 (PDT) Received: by mail-yb1-xb2d.google.com with SMTP id 3f1490d57ef6-b9a6eec8611so27529095276.0 for ; Mon, 08 May 2023 15:00:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583203; x=1686175203; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=tNF7BZ6TSMHS2fwAU3DRamWSillCVfAi2PAOw+/HR5A=; b=S/CZAdb4rcOzIupmMMK8ztoMdjZIPs5WXH1Fe1SO+T6fhIr4OdFwH73+GInCXZdR0H uyGTsrckc/YOyuAPOZf4y9T9xv/F72x5DbFhiK32YP6Uh66yLUB5E1gxvm80IXuCIWbY FtuCHcpTW7LM9BJaQEyjwbIz/7tOSgLj6WSF0A/AIJ0SxfNVDFTXfrO6ULqpD0cHlqYV QS0+CNymZvZwBpKuVbCE7gWemUcyjMjF+bQPqM4ujU30aTkMny1CW5OCWH6HBeAy9G9y kxINXhtgjSmUZMv+ddBsNUghToXeV6grJqeC3pp4yewOAkwJx7fRx6JEjumI7BwOmgCc 4FKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583203; x=1686175203; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=tNF7BZ6TSMHS2fwAU3DRamWSillCVfAi2PAOw+/HR5A=; b=NmO7jmSTyqxEpnSSQ0ipTqTerN1P2e/rgkrDZqPoXFaVI45jD88DhIMdjAuE43cdGW TnXDhN3m7CS2RNiHPfXWaIgZCyI3TrgFtQQ+MlXcIDn+wJLXjNxA8WuWLpymnozblsBi fFu0ISMuAwviPAZ5+qcBR3CK4jQm/+dYEGRSGORPJXHY8+pOxyK0YDlSJvmu2w8PQwOn Ot9CrF91ReFhylkn4WpOaRYP6brXHK0K/fz1MNK5Gtig+afg/XZ/6uIrY51twVuyLzYl QZmQEVzYLyAe4OQhcSDVL3z3GtbD12E7UiQ6APnUon0vO9WrBYRvSnZ73beijGP4CcZL LB+w== X-Gm-Message-State: AC+VfDxUvKdauIBOHB3+NDqcxkxC+rJrmlnaBY/HPrYm3HGyDLd9vbcl ItrnRmFdNb9jA4ECrRiYewgHIjluvpd0zkRoKNhEeg== X-Google-Smtp-Source: ACHHUZ6XSPImi6vB01v2th1SZp/SXS7nrIlPC4mBJOo6gumnev9Sxt8SXrBkpMz7zmIHEmMcMq3a9Q== X-Received: by 2002:a25:b189:0:b0:b8f:722b:3570 with SMTP id h9-20020a25b189000000b00b8f722b3570mr12623857ybj.3.1683583203400; Mon, 08 May 2023 15:00:03 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id k3-20020a25ed03000000b00b8f4916294csm2578057ybh.10.2023.05.08.15.00.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 15:00:03 -0700 (PDT) Date: Mon, 8 May 2023 18:00:01 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 07/15] refs: plumb `exclude_patterns` argument throughout Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The subsequent patch will want to access an optional `excluded_patterns` array within refs/packed-backend.c. To do so, the refs subsystem needs to be updated to pass this value across a number of different locations. Prepare for a future patch by introducing this plumbing now, passing NULLs at top-level APIs in order to make that patch less noisy and more easily readable. Signed-off-by: Taylor Blau --- ls-refs.c | 2 +- ref-filter.c | 5 +++-- refs.c | 32 +++++++++++++++++++------------- refs.h | 8 +++++++- refs/debug.c | 5 +++-- refs/files-backend.c | 5 +++-- refs/packed-backend.c | 5 +++-- refs/refs-internal.h | 7 ++++--- revision.c | 2 +- 9 files changed, 44 insertions(+), 27 deletions(-) diff --git a/ls-refs.c b/ls-refs.c index b9f3e08ec3..c7ad39611a 100644 --- a/ls-refs.c +++ b/ls-refs.c @@ -192,7 +192,7 @@ int ls_refs(struct repository *r, struct packet_reader *request) strvec_push(&data.prefixes, ""); refs_for_each_fullref_in_prefixes(get_main_ref_store(r), get_git_namespace(), data.prefixes.v, - send_ref, &data); + NULL, send_ref, &data); packet_fflush(stdout); strvec_clear(&data.prefixes); strbuf_release(&data.buf); diff --git a/ref-filter.c b/ref-filter.c index 93dc9b331f..c8ced1104b 100644 --- a/ref-filter.c +++ b/ref-filter.c @@ -2207,12 +2207,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter, if (!filter->name_patterns[0]) { /* no patterns; we have to look at everything */ - return for_each_fullref_in("", cb, cb_data); + return refs_for_each_fullref_in(get_main_ref_store(the_repository), + "", NULL, cb, cb_data); } return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository), NULL, filter->name_patterns, - cb, cb_data); + NULL, cb, cb_data); } /* diff --git a/refs.c b/refs.c index b9b77d2eff..538bde644e 100644 --- a/refs.c +++ b/refs.c @@ -1526,7 +1526,9 @@ int head_ref(each_ref_fn fn, void *cb_data) struct ref_iterator *refs_ref_iterator_begin( struct ref_store *refs, - const char *prefix, int trim, + const char *prefix, + const char **exclude_patterns, + int trim, enum do_for_each_ref_flags flags) { struct ref_iterator *iter; @@ -1542,8 +1544,7 @@ struct ref_iterator *refs_ref_iterator_begin( } } - iter = refs->be->iterator_begin(refs, prefix, flags); - + iter = refs->be->iterator_begin(refs, prefix, exclude_patterns, flags); /* * `iterator_begin()` already takes care of prefix, but we * might need to do some trimming: @@ -1577,7 +1578,7 @@ static int do_for_each_repo_ref(struct repository *r, const char *prefix, if (!refs) return 0; - iter = refs_ref_iterator_begin(refs, prefix, trim, flags); + iter = refs_ref_iterator_begin(refs, prefix, NULL, trim, flags); return do_for_each_repo_ref_iterator(r, iter, fn, cb_data); } @@ -1599,6 +1600,7 @@ static int do_for_each_ref_helper(struct repository *r, } static int do_for_each_ref(struct ref_store *refs, const char *prefix, + const char **exclude_patterns, each_ref_fn fn, int trim, enum do_for_each_ref_flags flags, void *cb_data) { @@ -1608,7 +1610,8 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix, if (!refs) return 0; - iter = refs_ref_iterator_begin(refs, prefix, trim, flags); + iter = refs_ref_iterator_begin(refs, prefix, exclude_patterns, trim, + flags); return do_for_each_repo_ref_iterator(the_repository, iter, do_for_each_ref_helper, &hp); @@ -1616,7 +1619,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix, int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data) { - return do_for_each_ref(refs, "", fn, 0, 0, cb_data); + return do_for_each_ref(refs, "", NULL, fn, 0, 0, cb_data); } int for_each_ref(each_ref_fn fn, void *cb_data) @@ -1627,7 +1630,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data) int refs_for_each_ref_in(struct ref_store *refs, const char *prefix, each_ref_fn fn, void *cb_data) { - return do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data); + return do_for_each_ref(refs, prefix, NULL, fn, strlen(prefix), 0, cb_data); } int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data) @@ -1638,13 +1641,14 @@ int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data) int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data) { return do_for_each_ref(get_main_ref_store(the_repository), - prefix, fn, 0, 0, cb_data); + prefix, NULL, fn, 0, 0, cb_data); } int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix, + const char **exclude_patterns, each_ref_fn fn, void *cb_data) { - return do_for_each_ref(refs, prefix, fn, 0, 0, cb_data); + return do_for_each_ref(refs, prefix, exclude_patterns, fn, 0, 0, cb_data); } int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data) @@ -1661,14 +1665,14 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data) int ret; strbuf_addf(&buf, "%srefs/", get_git_namespace()); ret = do_for_each_ref(get_main_ref_store(the_repository), - buf.buf, fn, 0, 0, cb_data); + buf.buf, NULL, fn, 0, 0, cb_data); strbuf_release(&buf); return ret; } int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data) { - return do_for_each_ref(refs, "", fn, 0, + return do_for_each_ref(refs, "", NULL, fn, 0, DO_FOR_EACH_INCLUDE_BROKEN, cb_data); } @@ -1738,6 +1742,7 @@ static void find_longest_prefixes(struct string_list *out, int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store, const char *namespace, const char **patterns, + const char **exclude_patterns, each_ref_fn fn, void *cb_data) { struct string_list prefixes = STRING_LIST_INIT_DUP; @@ -1753,7 +1758,8 @@ int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store, for_each_string_list_item(prefix, &prefixes) { strbuf_addstr(&buf, prefix->string); - ret = refs_for_each_fullref_in(ref_store, buf.buf, fn, cb_data); + ret = refs_for_each_fullref_in(ref_store, buf.buf, + exclude_patterns, fn, cb_data); if (ret) break; strbuf_setlen(&buf, namespace_len); @@ -2408,7 +2414,7 @@ int refs_verify_refname_available(struct ref_store *refs, strbuf_addstr(&dirname, refname + dirname.len); strbuf_addch(&dirname, '/'); - iter = refs_ref_iterator_begin(refs, dirname.buf, 0, + iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0, DO_FOR_EACH_INCLUDE_BROKEN); while ((ok = ref_iterator_advance(iter)) == ITER_OK) { if (skip && diff --git a/refs.h b/refs.h index 123cfa4424..d672d636cf 100644 --- a/refs.h +++ b/refs.h @@ -338,6 +338,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data); int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data); int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix, + const char **exclude_patterns, each_ref_fn fn, void *cb_data); int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data); @@ -345,10 +346,15 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data); * iterate all refs in "patterns" by partitioning patterns into disjoint sets * and iterating the longest-common prefix of each set. * + * references matching any pattern in "exclude_patterns" are omitted from the + * result set on a best-effort basis. + * * callers should be prepared to ignore references that they did not ask for. */ int refs_for_each_fullref_in_prefixes(struct ref_store *refs, - const char *namespace, const char **patterns, + const char *namespace, + const char **patterns, + const char **exclude_patterns, each_ref_fn fn, void *cb_data); /** diff --git a/refs/debug.c b/refs/debug.c index adc34c836f..8131133e99 100644 --- a/refs/debug.c +++ b/refs/debug.c @@ -228,11 +228,12 @@ static struct ref_iterator_vtable debug_ref_iterator_vtable = { static struct ref_iterator * debug_ref_iterator_begin(struct ref_store *ref_store, const char *prefix, - unsigned int flags) + const char **exclude_patterns, unsigned int flags) { struct debug_ref_store *drefs = (struct debug_ref_store *)ref_store; struct ref_iterator *res = - drefs->refs->be->iterator_begin(drefs->refs, prefix, flags); + drefs->refs->be->iterator_begin(drefs->refs, prefix, + exclude_patterns, flags); struct debug_ref_iterator *diter = xcalloc(1, sizeof(*diter)); base_ref_iterator_init(&diter->base, &debug_ref_iterator_vtable, 1); diter->iter = res; diff --git a/refs/files-backend.c b/refs/files-backend.c index d0581ee41a..5fae864334 100644 --- a/refs/files-backend.c +++ b/refs/files-backend.c @@ -827,7 +827,8 @@ static struct ref_iterator_vtable files_ref_iterator_vtable = { static struct ref_iterator *files_ref_iterator_begin( struct ref_store *ref_store, - const char *prefix, unsigned int flags) + const char *prefix, const char **exclude_patterns, + unsigned int flags) { struct files_ref_store *refs; struct ref_iterator *loose_iter, *packed_iter, *overlay_iter; @@ -872,7 +873,7 @@ static struct ref_iterator *files_ref_iterator_begin( * the packed and loose references. */ packed_iter = refs_ref_iterator_begin( - refs->packed_ref_store, prefix, 0, + refs->packed_ref_store, prefix, exclude_patterns, 0, DO_FOR_EACH_INCLUDE_BROKEN); overlay_iter = overlay_ref_iterator_begin(loose_iter, packed_iter); diff --git a/refs/packed-backend.c b/refs/packed-backend.c index 34c0c4e20f..e54e78e540 100644 --- a/refs/packed-backend.c +++ b/refs/packed-backend.c @@ -923,7 +923,8 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = { static struct ref_iterator *packed_ref_iterator_begin( struct ref_store *ref_store, - const char *prefix, unsigned int flags) + const char *prefix, const char **exclude_patterns, + unsigned int flags) { struct packed_ref_store *refs; struct snapshot *snapshot; @@ -1148,7 +1149,7 @@ static int write_with_updates(struct packed_ref_store *refs, * list of refs is exhausted, set iter to NULL. When the list * of updates is exhausted, leave i set to updates->nr. */ - iter = packed_ref_iterator_begin(&refs->base, "", + iter = packed_ref_iterator_begin(&refs->base, "", NULL, DO_FOR_EACH_INCLUDE_BROKEN); if ((ok = ref_iterator_advance(iter)) != ITER_OK) iter = NULL; diff --git a/refs/refs-internal.h b/refs/refs-internal.h index a85d113123..28a11b9d61 100644 --- a/refs/refs-internal.h +++ b/refs/refs-internal.h @@ -367,8 +367,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator); */ struct ref_iterator *refs_ref_iterator_begin( struct ref_store *refs, - const char *prefix, int trim, - enum do_for_each_ref_flags flags); + const char *prefix, const char **exclude_patterns, + int trim, enum do_for_each_ref_flags flags); /* * A callback function used to instruct merge_ref_iterator how to @@ -570,7 +570,8 @@ typedef int copy_ref_fn(struct ref_store *ref_store, */ typedef struct ref_iterator *ref_iterator_begin_fn( struct ref_store *ref_store, - const char *prefix, unsigned int flags); + const char *prefix, const char **exclude_patterns, + unsigned int flags); /* reflog functions */ diff --git a/revision.c b/revision.c index b33cc1d106..89953592f9 100644 --- a/revision.c +++ b/revision.c @@ -2670,7 +2670,7 @@ static int for_each_bisect_ref(struct ref_store *refs, each_ref_fn fn, struct strbuf bisect_refs = STRBUF_INIT; int status; strbuf_addf(&bisect_refs, "refs/bisect/%s", term); - status = refs_for_each_fullref_in(refs, bisect_refs.buf, fn, cb_data); + status = refs_for_each_fullref_in(refs, bisect_refs.buf, NULL, fn, cb_data); strbuf_release(&bisect_refs); return status; } From patchwork Mon May 8 22:00:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C64A1C77B7F for ; Mon, 8 May 2023 22:00:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234508AbjEHWAb (ORCPT ); Mon, 8 May 2023 18:00:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40118 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234503AbjEHWAP (ORCPT ); Mon, 8 May 2023 18:00:15 -0400 Received: from mail-yw1-x112c.google.com (mail-yw1-x112c.google.com [IPv6:2607:f8b0:4864:20::112c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C8C8F902A for ; Mon, 8 May 2023 15:00:07 -0700 (PDT) Received: by mail-yw1-x112c.google.com with SMTP id 00721157ae682-55cc8aadc97so76718147b3.3 for ; Mon, 08 May 2023 15:00:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583206; x=1686175206; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=AQdeURv9Ds0ZzUIXG/mLerajMScjALRyBD6QMAArqy8=; b=EaqcsrAZXXtUWaUy7W5IppN5b+PZRkXcFP4qJ70Kyy6Lf23TmzGmXcQfQ870N7pD12 dfXr0p1FBJ+p7Ov3M8bg8fXnQdOtXQWJiMiOIGZy8t/m1ELqhHX1fdfWv2kzI6Kz7a8E S41GtKGfywQHwIKy/+F+tSrAoYKcLLfqOVe78x3O0m44sE2woVMs+RgiLO0XKXppw2xh Lk/kDt+GTbQOwOy3n9kn8ePSKTaRvr2itqXY79M59TR7VYXMHECpuXodeMtyW9mY54DX HjQNKMNspFfKR7ewck4ctsAeaJFo5tnHscuuoc1AGVJSv1byS/CXSAXKj/qqnjC1HaNx Y8bA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583206; x=1686175206; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=AQdeURv9Ds0ZzUIXG/mLerajMScjALRyBD6QMAArqy8=; b=C0/m0nXyIJBBtObbxglf+Aw/1VfO2kuhyMh0mwkCZV/2ZWgbZXy4mpFQhsmkHunavb NXRypmrgEYV9Of7j82OHV8JmVfjMig4Kgm8PqVJSpP8yROMTq0JaTI9+pLqERpPKnZXI ps2Sz1mjvf3dtwI4nvXjr+BsCPNRTwCFLuJ3xion+8abZyu9BhD94aR6fw0ST4UAuSMJ S2AwjNow2NJscMc8/FFgWAtuSV/GHMXZsBcZaMfhKPImOL8P84a8DLrI0HI0ydrQYvvt uNslTpSXDHBJ7TCTD70hoUq0nTuZCZFZjqj8TkrUdekKMcs7NbBPkK2t5VWg0BC8P8xM jzSA== X-Gm-Message-State: AC+VfDxuRHoHI05UxclZxl304XBuoF0lInKfPODnAq612c9PDFNZjdZl TuYgJw4VHOT21OGl4rcaqljFJqTlfDkJNnWQWK/WmA== X-Google-Smtp-Source: ACHHUZ4sc5XWO7+M/wHel5YQgkhnCzCVm6s176LVg46dt/OTf+aXtNPsdxP58GTOH8KNx4kIBbshcQ== X-Received: by 2002:a0d:e853:0:b0:55c:2511:b30c with SMTP id r80-20020a0de853000000b0055c2511b30cmr13119834ywe.5.1683583206662; Mon, 08 May 2023 15:00:06 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id u205-20020a8147d6000000b00545a08184cesm2779077ywa.94.2023.05.08.15.00.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 15:00:06 -0700 (PDT) Date: Mon, 8 May 2023 18:00:05 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 08/15] refs/packed-backend.c: refactor `find_reference_location()` Message-ID: <836a5665b7df065811edc678cb8e70004f7b7c49.1683581621.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The function `find_reference_location()` is used to perform a binary search-like function over the contents of a repository's `$GIT_DIR/packed-refs` file. The search it implements is unlike a standard binary search in that the records it searches over are not of a fixed width, so the comparison must locate the end of a record before comparing it. Extract the core routine of `find_reference_location()` in order to implement a function in the following patch which will find the first location in the `packed-refs` file that *doesn't* match the given pattern. The behavior of `find_reference_location()` is unchanged. Co-authored-by: Jeff King Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- refs/packed-backend.c | 46 +++++++++++++++++++++++++------------------ 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/refs/packed-backend.c b/refs/packed-backend.c index e54e78e540..98f96bf3ee 100644 --- a/refs/packed-backend.c +++ b/refs/packed-backend.c @@ -302,7 +302,8 @@ static int cmp_packed_ref_records(const void *v1, const void *v2) * Compare a snapshot record at `rec` to the specified NUL-terminated * refname. */ -static int cmp_record_to_refname(const char *rec, const char *refname) +static int cmp_record_to_refname(const char *rec, const char *refname, + int start) { const char *r1 = rec + the_hash_algo->hexsz + 1; const char *r2 = refname; @@ -311,7 +312,7 @@ static int cmp_record_to_refname(const char *rec, const char *refname) if (*r1 == '\n') return *r2 ? -1 : 0; if (!*r2) - return 1; + return start ? 1 : -1; if (*r1 != *r2) return (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1; r1++; @@ -526,22 +527,9 @@ static int load_contents(struct snapshot *snapshot) return 1; } -/* - * Find the place in `snapshot->buf` where the start of the record for - * `refname` starts. If `mustexist` is true and the reference doesn't - * exist, then return NULL. If `mustexist` is false and the reference - * doesn't exist, then return the point where that reference would be - * inserted, or `snapshot->eof` (which might be NULL) if it would be - * inserted at the end of the file. In the latter mode, `refname` - * doesn't have to be a proper reference name; for example, one could - * search for "refs/replace/" to find the start of any replace - * references. - * - * The record is sought using a binary search, so `snapshot->buf` must - * be sorted. - */ -static const char *find_reference_location(struct snapshot *snapshot, - const char *refname, int mustexist) +static const char *find_reference_location_1(struct snapshot *snapshot, + const char *refname, int mustexist, + int start) { /* * This is not *quite* a garden-variety binary search, because @@ -571,7 +559,7 @@ static const char *find_reference_location(struct snapshot *snapshot, mid = lo + (hi - lo) / 2; rec = find_start_of_record(lo, mid); - cmp = cmp_record_to_refname(rec, refname); + cmp = cmp_record_to_refname(rec, refname, start); if (cmp < 0) { lo = find_end_of_record(mid, hi); } else if (cmp > 0) { @@ -587,6 +575,26 @@ static const char *find_reference_location(struct snapshot *snapshot, return lo; } +/* + * Find the place in `snapshot->buf` where the start of the record for + * `refname` starts. If `mustexist` is true and the reference doesn't + * exist, then return NULL. If `mustexist` is false and the reference + * doesn't exist, then return the point where that reference would be + * inserted, or `snapshot->eof` (which might be NULL) if it would be + * inserted at the end of the file. In the latter mode, `refname` + * doesn't have to be a proper reference name; for example, one could + * search for "refs/replace/" to find the start of any replace + * references. + * + * The record is sought using a binary search, so `snapshot->buf` must + * be sorted. + */ +static const char *find_reference_location(struct snapshot *snapshot, + const char *refname, int mustexist) +{ + return find_reference_location_1(snapshot, refname, mustexist, 1); +} + /* * Create a newly-allocated `snapshot` of the `packed-refs` file in * its current state and return it. The return value will already have From patchwork Mon May 8 22:00:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C2B2C77B7F for ; Mon, 8 May 2023 22:00:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234444AbjEHWAk (ORCPT ); Mon, 8 May 2023 18:00:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234485AbjEHWAV (ORCPT ); Mon, 8 May 2023 18:00:21 -0400 Received: from mail-yw1-x112f.google.com (mail-yw1-x112f.google.com [IPv6:2607:f8b0:4864:20::112f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4802A59F9 for ; Mon, 8 May 2023 15:00:11 -0700 (PDT) Received: by mail-yw1-x112f.google.com with SMTP id 00721157ae682-559ded5e170so75048617b3.3 for ; Mon, 08 May 2023 15:00:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583210; x=1686175210; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=LgGZeyce8W0QZZ370IlNHcF36abGDFrZnbfxGCwVQUs=; b=Tw9oTmAreb62qLHwaIwKYlooGElV4jwn357KjXRuyxulTf1Clq7oG2Lh0/4uOBjvrj ADu6ur4OH3r+J56+plsQMs+1XSoQgoB/Kfk8D/eucmYTRrqcLuPYfhBjVbXVUmBS5PM6 +PfZ1+2wrO904QJEL1QqmIeeRZxxZl0BHIcXmiq5RWj667sbOzHClivLFDvn7XQAmU7E xgXxBtjkG+G3Yptzd0zpXioxeXzLQo6oAjD82t9zfydn5+EfmAwOoU+oVDWV2eBEL1Zr R52Vd+kR7N1po2dMB+Llg3Ty+tbo1p89DK6xPsk9reH8Z9lOjIJRUdCc7zLLXm5+3pr0 X+OQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583210; x=1686175210; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LgGZeyce8W0QZZ370IlNHcF36abGDFrZnbfxGCwVQUs=; b=DUTkWpw0ApsF5f5mRHGDUmd6Y9jgLGpRStxj4TT3ZLxxTS76QE71zCIrNYu0lJGB9E +RwaFzMQ7Xkvx97kmGldujrhSgiA8TYce5IjQQuPl1aPlLIzfAFRG1QMPLSmf166TDdQ CK+vCtM1XTLkMdPEPZTRmuPZNU4fI0EaHk+nTdRIadbs4INFBFTWNJuClP/eF4Cs2wIA ReIGbeE41Tg6CfVxQm1w3MAUmFUPFdFaueex1XSKcQCF7CWCef75fy7WPPGE+aohmwF9 oEQ2HT0pNjZehR256we94PUEg5p2fr7ekpqtV0YBXrKZUQiUwTQTYATLEldp5gds9heS tY+A== X-Gm-Message-State: AC+VfDxhTEcZEEbXUfOmhkHHs96e+dl/6jRv6Pk4qqH2TbvPomMeRVY3 sXTRlgQd6U09i8CoPW0J5r9j1bL2Xou30zOQqrD4JQ== X-Google-Smtp-Source: ACHHUZ7QAX1uGqSw+C3geJY2B/GN0Fq7oEOnYHz4GrmwLPuPFabo4b88ys41GPEGZW60ZUcXVu+T8A== X-Received: by 2002:a81:6d05:0:b0:55a:c51:9a15 with SMTP id i5-20020a816d05000000b0055a0c519a15mr13279148ywc.22.1683583210147; Mon, 08 May 2023 15:00:10 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id n12-20020a819c4c000000b005460251b0d9sm2783092ywa.82.2023.05.08.15.00.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 15:00:09 -0700 (PDT) Date: Mon, 8 May 2023 18:00:08 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s) Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When iterating through the `packed-refs` file in order to answer a query like: $ git for-each-ref --exclude=refs/__hidden__ it would be useful to avoid walking over all of the entries in `refs/__hidden__/*` when possible, since we know that the ref-filter code is going to throw them away anyways. In certain circumstances, doing so is possible. The algorithm for doing so is as follows: - For each excluded pattern, find the first record that matches it, and the first pattern that *doesn't* match it (i.e. the location you'd next want to consider when excluding that pattern). - Sort the patterns by their starting location within the `packed-refs` file. - Construct a skip list of regions by combining adjacent and overlapping regions from the previous step. - When iterating through the `packed-refs` file, if `iter->pos` is ever contained in one of the regions from the previous steps, advance `iter->pos` past the end of that region, and continue enumeration. Note that this optimization is only possible when none of the excluded pattern(s) have special meta-characters in them. To see why this is the case, consider the exclusion pattern "refs/foo[a]". In general, in order to find the location of the first record that matches this pattern, we could only consider up to the first meta-character, "refs/foo". But this doesn't work, since the excluded region we'd come up with would include "refs/foobar", even though it is not excluded. There are a few other gotchas worth considering. First, note that the skip list is sorted, so once we skip past a region, we can avoid considering it (or any regions preceding it) again. The member `skip_pos` is used to track the first next-possible region to jump through. Second, note that the exclusion list is best-effort, since we do not handle loose references, and because of the meta-character issue above. In repositories with a large number of hidden references, the speed-up can be significant. Tests here are done with a copy of linux.git with a reference "refs/pull/N" pointing at every commit, as in: $ git rev-list HEAD | awk '{ print "create refs/pull/" NR " " $0 }' | git update-ref --stdin $ git pack-refs --all , it is significantly faster to have `for-each-ref` skip over the excluded references, as opposed to filtering them out after the fact: $ hyperfine \ 'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"' \ 'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' Benchmark 1: git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/" Time (mean ± σ): 802.7 ms ± 2.1 ms [User: 691.6 ms, System: 147.0 ms] Range (min … max): 800.0 ms … 807.7 ms 10 runs Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull" Time (mean ± σ): 4.7 ms ± 0.3 ms [User: 0.7 ms, System: 4.0 ms] Range (min … max): 4.3 ms … 6.7 ms 422 runs Summary 'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' ran 172.03 ± 9.60 times faster than 'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"' Using the skip list is fairly straightforward (see the changes to `refs/packed-backend.c::next_record()`), but constructing the list is not. To ensure that the construction is correct, add a new suite of tests in t1419 covering various corner cases (overlapping regions, partially overlapping regions, adjacent regions, etc.). Co-authored-by: Jeff King Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- ref-filter.c | 5 +- refs/packed-backend.c | 150 +++++++++++++++++++++++++++++++++++++- t/helper/test-ref-store.c | 10 +++ t/t1419-exclude-refs.sh | 101 +++++++++++++++++++++++++ 4 files changed, 263 insertions(+), 3 deletions(-) create mode 100755 t/t1419-exclude-refs.sh diff --git a/ref-filter.c b/ref-filter.c index c8ced1104b..56ebd332fa 100644 --- a/ref-filter.c +++ b/ref-filter.c @@ -2208,12 +2208,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter, if (!filter->name_patterns[0]) { /* no patterns; we have to look at everything */ return refs_for_each_fullref_in(get_main_ref_store(the_repository), - "", NULL, cb, cb_data); + "", filter->exclude.v, cb, cb_data); } return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository), NULL, filter->name_patterns, - NULL, cb, cb_data); + filter->exclude.v, + cb, cb_data); } /* diff --git a/refs/packed-backend.c b/refs/packed-backend.c index 98f96bf3ee..137a4233f6 100644 --- a/refs/packed-backend.c +++ b/refs/packed-backend.c @@ -595,6 +595,21 @@ static const char *find_reference_location(struct snapshot *snapshot, return find_reference_location_1(snapshot, refname, mustexist, 1); } +/* + * Find the place in `snapshot->buf` after the end of the record for + * `refname`. In other words, find the location of first thing *after* + * `refname`. + * + * Other semantics are identical to the ones in + * `find_reference_location()`. + */ +static const char *find_reference_location_end(struct snapshot *snapshot, + const char *refname, + int mustexist) +{ + return find_reference_location_1(snapshot, refname, mustexist, 0); +} + /* * Create a newly-allocated `snapshot` of the `packed-refs` file in * its current state and return it. The return value will already have @@ -786,6 +801,13 @@ struct packed_ref_iterator { /* The end of the part of the buffer that will be iterated over: */ const char *eof; + struct skip_list_entry { + const char *start; + const char *end; + } *skip; + size_t skip_nr, skip_alloc; + size_t skip_pos; + /* Scratch space for current values: */ struct object_id oid, peeled; struct strbuf refname_buf; @@ -803,14 +825,34 @@ struct packed_ref_iterator { */ static int next_record(struct packed_ref_iterator *iter) { - const char *p = iter->pos, *eol; + const char *p, *eol; strbuf_reset(&iter->refname_buf); + /* + * If iter->pos is contained within a skipped region, jump past + * it. + * + * Note that each skipped region is considered at most once, + * since they are ordered based on their starting position. + */ + while (iter->skip_pos < iter->skip_nr) { + struct skip_list_entry *curr = &iter->skip[iter->skip_pos]; + if (iter->pos < curr->start) + break; /* not to the next jump yet */ + + iter->skip_pos++; + if (iter->pos < curr->end) { + iter->pos = curr->end; + break; + } + } + if (iter->pos == iter->eof) return ITER_DONE; iter->base.flags = REF_ISPACKED; + p = iter->pos; if (iter->eof - p < the_hash_algo->hexsz + 2 || parse_oid_hex(p, &iter->oid, &p) || @@ -918,6 +960,7 @@ static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator) int ok = ITER_DONE; strbuf_release(&iter->refname_buf); + free(iter->skip); release_snapshot(iter->snapshot); base_ref_iterator_free(ref_iterator); return ok; @@ -929,6 +972,108 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = { .abort = packed_ref_iterator_abort }; +static int skip_list_entry_cmp(const void *va, const void *vb) +{ + const struct skip_list_entry *a = va; + const struct skip_list_entry *b = vb; + + if (a->start < b->start) + return -1; + if (a->start > b->start) + return 1; + return 0; +} + +static int has_glob_special(const char *str) +{ + const char *p; + for (p = str; *p; p++) { + if (is_glob_special(*p)) + return 1; + } + return 0; +} + +static const char *ptr_max(const char *x, const char *y) +{ + if (x > y) + return x; + return y; +} + +static void populate_excluded_skip_list(struct packed_ref_iterator *iter, + struct snapshot *snapshot, + const char **excluded_patterns) +{ + size_t i, j; + const char **pattern; + + if (!excluded_patterns) + return; + + for (pattern = excluded_patterns; *pattern; pattern++) { + struct skip_list_entry *e; + + /* + * We can't feed any excludes with globs in them to the + * refs machinery. It only understands prefix matching. + * We likewise can't even feed the string leading up to + * the first meta-character, as something like "foo[a]" + * should not exclude "foobar" (but the prefix "foo" + * would match that and mark it for exclusion). + */ + if (has_glob_special(*pattern)) + continue; + + ALLOC_GROW(iter->skip, iter->skip_nr + 1, iter->skip_alloc); + + e = &iter->skip[iter->skip_nr++]; + e->start = find_reference_location(snapshot, *pattern, 0); + e->end = find_reference_location_end(snapshot, *pattern, 0); + } + + if (!iter->skip_nr) { + /* + * Every entry in exclude_patterns has a meta-character, + * nothing to do here. + */ + return; + } + + QSORT(iter->skip, iter->skip_nr, skip_list_entry_cmp); + + /* + * As an optimization, merge adjacent entries in the skip list + * to jump forwards as far as possible when entering a skipped + * region. + * + * For example, if we have two skipped regions: + * + * [[A, B], [B, C]] + * + * we want to combine that into a single entry jumping from A to + * C. + */ + for (i = 1, j = 1; i < iter->skip_nr; i++) { + struct skip_list_entry *ours = &iter->skip[i]; + struct skip_list_entry *prev = &iter->skip[i - 1]; + + if (ours->start == ours->end) { + /* ignore empty regions (no matching entries) */ + continue; + } else if (prev->end >= ours->start) { + /* overlapping regions extend the previous one */ + prev->end = ptr_max(prev->end, ours->end); + } else { + /* otherwise, insert a new region */ + iter->skip[j++] = *ours; + } + } + + iter->skip_nr = j; + iter->skip_pos = 0; +} + static struct ref_iterator *packed_ref_iterator_begin( struct ref_store *ref_store, const char *prefix, const char **exclude_patterns, @@ -964,6 +1109,9 @@ static struct ref_iterator *packed_ref_iterator_begin( ref_iterator = &iter->base; base_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable, 1); + if (exclude_patterns) + populate_excluded_skip_list(iter, snapshot, exclude_patterns); + iter->snapshot = snapshot; acquire_snapshot(snapshot); diff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c index 6d8f844e9c..2bff003f7c 100644 --- a/t/helper/test-ref-store.c +++ b/t/helper/test-ref-store.c @@ -175,6 +175,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv) return refs_for_each_ref_in(refs, prefix, each_ref, NULL); } +static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv) +{ + const char *prefix = notnull(*argv++, "prefix"); + const char **exclude_patterns = argv; + + return refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref, + NULL); +} + static int cmd_resolve_ref(struct ref_store *refs, const char **argv) { struct object_id oid = *null_oid(); @@ -307,6 +316,7 @@ static struct command commands[] = { { "delete-refs", cmd_delete_refs }, { "rename-ref", cmd_rename_ref }, { "for-each-ref", cmd_for_each_ref }, + { "for-each-ref--exclude", cmd_for_each_ref__exclude }, { "resolve-ref", cmd_resolve_ref }, { "verify-ref", cmd_verify_ref }, { "for-each-reflog", cmd_for_each_reflog }, diff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh new file mode 100755 index 0000000000..da5265a5a8 --- /dev/null +++ b/t/t1419-exclude-refs.sh @@ -0,0 +1,101 @@ +#!/bin/sh + +test_description='test exclude_patterns functionality in main ref store' + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +TEST_PASSES_SANITIZE_LEAK=true +. ./test-lib.sh + +for_each_ref__exclude () { + test-tool ref-store main for-each-ref--exclude "$@" >actual.raw + cut -d ' ' -f 2 actual.raw +} + +for_each_ref () { + git for-each-ref --format='%(refname)' "$@" +} + +test_expect_success 'setup' ' + test_commit --no-tag base && + base="$(git rev-parse HEAD)" && + + for name in foo bar baz quux + do + for i in 1 2 3 + do + echo "create refs/heads/$name/$i $base" || return 1 + done || return 1 + done >in && + echo "delete refs/heads/main" >>in && + + git update-ref --stdin actual && + for_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect && + + test_cmp expect actual +' + +test_expect_success 'for_each_ref__exclude(refs/heads/bar/)' ' + # region at beginning + for_each_ref__exclude refs/heads refs/heads/bar >actual && + for_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect && + + test_cmp expect actual +' + +test_expect_success 'for_each_ref__exclude(refs/heads/quux/)' ' + # region at end + for_each_ref__exclude refs/heads refs/heads/quux >actual && + for_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect && + + test_cmp expect actual +' + +test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/quux/)' ' + # disjoint regions + for_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual && + for_each_ref refs/heads/baz refs/heads/foo >expect && + + test_cmp expect actual +' + +test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/baz/)' ' + # adjacent, non-overlapping regions + for_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual && + for_each_ref refs/heads/foo refs/heads/quux >expect && + + test_cmp expect actual +' + +test_expect_success 'for_each_ref__exclude(refs/heads/ba refs/heads/baz/)' ' + # overlapping region + for_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual && + for_each_ref refs/heads/foo refs/heads/quux >expect && + + test_cmp expect actual +' + +test_expect_success 'for_each_ref__exclude(refs/heads/does/not/exist)' ' + # empty region + for_each_ref__exclude refs/heads refs/heads/does/not/exist >actual && + for_each_ref >expect && + + test_cmp expect actual +' + +test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' ' + # discards meta-characters + for_each_ref__exclude refs/heads "refs/heads/ba*" >actual && + for_each_ref >expect && + + test_cmp expect actual +' + +test_done From patchwork Mon May 8 22:00:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B35FBC7EE22 for ; Mon, 8 May 2023 22:00:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234493AbjEHWAl (ORCPT ); Mon, 8 May 2023 18:00:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40354 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234487AbjEHWAV (ORCPT ); Mon, 8 May 2023 18:00:21 -0400 Received: from mail-yb1-xb2d.google.com (mail-yb1-xb2d.google.com [IPv6:2607:f8b0:4864:20::b2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE46D4EC1 for ; Mon, 8 May 2023 15:00:13 -0700 (PDT) Received: by mail-yb1-xb2d.google.com with SMTP id 3f1490d57ef6-b9a6eec8611so27529868276.0 for ; Mon, 08 May 2023 15:00:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583213; x=1686175213; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=wXulJv0GbS2RuuPN7kYulGtzj0vZAeVDDYhxBrfWDNs=; b=SiwDKqUhCga4bpvU09mZeV0Oflkteziys76cM2/uNTTPXzVWYqt5iJ6NmnqpxYHaQz XnwSydEKvFNnpWY2l4a6ByyRF4EiDl696P5PijyTLdwENkbqTateoBQt/JDmBmMg7oU+ pVkRldw6+08+T/IRcZgR7uHUlGNHVniyu4yaQwhQu5gr+ToGeAAZvUmJX+A9+r9RuvQY XMwIy2TALq41d2PglHCHAt8mW5y9lg3w5mjo3Z+rbjOb9v5zrXsTQqp5qq5P7C/NDF6z TLEKYhWu3X9mj1OWPAoPRtkiVtu68zQP7jpslvefDYC/PZr839LRGaXplqf/L+g4Vm9V uK8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583213; x=1686175213; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=wXulJv0GbS2RuuPN7kYulGtzj0vZAeVDDYhxBrfWDNs=; b=Ht5i/oTa8Iv7Qx7/YBm1kGFuQr1khSA3VOzExNyXnpaew7Bm8Dux9Ax0vFlcPE4DB+ OdEWyLKXS6ODYXtmudTkg9a8wc0X9K4Hrg53il2jd4ehomErG+FYrb6aAzpiuGYDCn5D ARCdeq5o6IHV3zYmGNbytHWIwAmqZw8WV7+ikXcXE2IevvGTUzRmXNMVssPH3gBHErel S0K1ZnGJhWtxvNGa+0DsFzGB1I5WutFLgaf+xMegrukQkhSQPLj+mR4ucftg7X+YZ/o7 bJl2XB6pRgZuKEJrH0rklPtxvKjjUfUxk1C7VdB7s4PQ8h3gwPv3ahQNhsX9fKiV6tvj 6ftQ== X-Gm-Message-State: AC+VfDzAlgh9kqfQ3wkbHpv946Pe5CsF6QjrZfanXYWJxETT6KnVM4ge NklfSucMQ4wS34YLqey81ydetQIvLe8moS9HjiiTTA== X-Google-Smtp-Source: ACHHUZ4pDWhX0g+OSqNpvUA1t6NLq9gAMrxWqmRpu/B8G1he2KMYFf6/wjGWBN2nMdTDG4ds7vURkA== X-Received: by 2002:a0d:ea0c:0:b0:550:65b4:ca60 with SMTP id t12-20020a0dea0c000000b0055065b4ca60mr13946504ywe.8.1683583213394; Mon, 08 May 2023 15:00:13 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id g127-20020a0df685000000b0054fba955474sm2816978ywf.17.2023.05.08.15.00.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 15:00:13 -0700 (PDT) Date: Mon, 8 May 2023 18:00:11 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 10/15] refs/packed-backend.c: add trace2 counters for skip list Message-ID: <5698c2794f8c780b4f8d8591ceb4475ec26d17c6.1683581621.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The previous commit added low-level tests to ensure that the packed-refs iterator did not enumerate excluded sections of the refspace. However, there was no guarantee that these sections weren't being visited, only that they were being suppressed from the output. To harden these tests, add a trace2 counter which tracks the number of regions skipped by the packed-refs iterator, and assert on its value. Suggested-by: Derrick Stolee Signed-off-by: Taylor Blau --- refs/packed-backend.c | 2 ++ t/t1419-exclude-refs.sh | 54 ++++++++++++++++++++++++++++------------- trace2.h | 2 ++ trace2/tr2_ctr.c | 5 ++++ 4 files changed, 46 insertions(+), 17 deletions(-) diff --git a/refs/packed-backend.c b/refs/packed-backend.c index 137a4233f6..ddfa9add14 100644 --- a/refs/packed-backend.c +++ b/refs/packed-backend.c @@ -11,6 +11,7 @@ #include "../chdir-notify.h" #include "../wrapper.h" #include "../write-or-die.h" +#include "../trace2.h" enum mmap_strategy { /* @@ -844,6 +845,7 @@ static int next_record(struct packed_ref_iterator *iter) iter->skip_pos++; if (iter->pos < curr->end) { iter->pos = curr->end; + trace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_SKIPS, 1); break; } } diff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh index da5265a5a8..051b5a54ce 100755 --- a/t/t1419-exclude-refs.sh +++ b/t/t1419-exclude-refs.sh @@ -9,7 +9,8 @@ TEST_PASSES_SANITIZE_LEAK=true . ./test-lib.sh for_each_ref__exclude () { - test-tool ref-store main for-each-ref--exclude "$@" >actual.raw + GIT_TRACE2_PERF=1 test-tool ref-store main \ + for-each-ref--exclude "$@" >actual.raw cut -d ' ' -f 2 actual.raw } @@ -17,6 +18,17 @@ for_each_ref () { git for-each-ref --format='%(refname)' "$@" } +assert_skips () { + local nr="$1" + local trace="$2" + + grep -q "name:skips_made value:$nr" $trace +} + +assert_no_skips () { + ! assert_skips ".*" "$1" +} + test_expect_success 'setup' ' test_commit --no-tag base && base="$(git rev-parse HEAD)" && @@ -36,66 +48,74 @@ test_expect_success 'setup' ' test_expect_success 'for_each_ref__exclude(refs/heads/foo/)' ' # region in middle - for_each_ref__exclude refs/heads refs/heads/foo >actual && + for_each_ref__exclude refs/heads refs/heads/foo >actual 2>perf && for_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect && - test_cmp expect actual + test_cmp expect actual && + assert_skips 1 perf ' test_expect_success 'for_each_ref__exclude(refs/heads/bar/)' ' # region at beginning - for_each_ref__exclude refs/heads refs/heads/bar >actual && + for_each_ref__exclude refs/heads refs/heads/bar >actual 2>perf && for_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect && - test_cmp expect actual + test_cmp expect actual && + assert_skips 1 perf ' test_expect_success 'for_each_ref__exclude(refs/heads/quux/)' ' # region at end - for_each_ref__exclude refs/heads refs/heads/quux >actual && + for_each_ref__exclude refs/heads refs/heads/quux >actual 2>perf && for_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect && - test_cmp expect actual + test_cmp expect actual && + assert_skips 1 perf ' test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/quux/)' ' # disjoint regions - for_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual && + for_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual 2>perf && for_each_ref refs/heads/baz refs/heads/foo >expect && - test_cmp expect actual + test_cmp expect actual && + assert_skips 2 perf ' test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/baz/)' ' # adjacent, non-overlapping regions - for_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual && + for_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual 2>perf && for_each_ref refs/heads/foo refs/heads/quux >expect && - test_cmp expect actual + test_cmp expect actual && + assert_skips 1 perf ' test_expect_success 'for_each_ref__exclude(refs/heads/ba refs/heads/baz/)' ' # overlapping region - for_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual && + for_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual 2>perf && for_each_ref refs/heads/foo refs/heads/quux >expect && - test_cmp expect actual + test_cmp expect actual && + assert_skips 1 perf ' test_expect_success 'for_each_ref__exclude(refs/heads/does/not/exist)' ' # empty region - for_each_ref__exclude refs/heads refs/heads/does/not/exist >actual && + for_each_ref__exclude refs/heads refs/heads/does/not/exist >actual 2>perf && for_each_ref >expect && - test_cmp expect actual + test_cmp expect actual && + assert_no_skips ' test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' ' # discards meta-characters - for_each_ref__exclude refs/heads "refs/heads/ba*" >actual && + for_each_ref__exclude refs/heads "refs/heads/ba*" >actual 2>perf && for_each_ref >expect && - test_cmp expect actual + test_cmp expect actual && + assert_no_skips ' test_done diff --git a/trace2.h b/trace2.h index 4ced30c0db..6a116f60a9 100644 --- a/trace2.h +++ b/trace2.h @@ -551,6 +551,8 @@ enum trace2_counter_id { TRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */ TRACE2_COUNTER_ID_TEST2, /* emits summary and thread events */ + TRACE2_COUNTER_ID_PACKED_REFS_SKIPS, /* counts number of skips */ + /* Add additional counter definitions before here. */ TRACE2_NUMBER_OF_COUNTERS }; diff --git a/trace2/tr2_ctr.c b/trace2/tr2_ctr.c index b342d3b1a3..f7efbc7646 100644 --- a/trace2/tr2_ctr.c +++ b/trace2/tr2_ctr.c @@ -27,6 +27,11 @@ static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTER .name = "test2", .want_per_thread_events = 1, }, + [TRACE2_COUNTER_ID_PACKED_REFS_SKIPS] = { + .category = "packed-refs", + .name = "skips_made", + .want_per_thread_events = 0, + }, /* Add additional metadata before here. */ }; From patchwork Mon May 8 22:00:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBC88C77B7F for ; Mon, 8 May 2023 22:00:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234458AbjEHWAx (ORCPT ); Mon, 8 May 2023 18:00:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40226 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234522AbjEHWA3 (ORCPT ); Mon, 8 May 2023 18:00:29 -0400 Received: from mail-yb1-xb2f.google.com (mail-yb1-xb2f.google.com [IPv6:2607:f8b0:4864:20::b2f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2DEC24EFC for ; Mon, 8 May 2023 15:00:24 -0700 (PDT) Received: by mail-yb1-xb2f.google.com with SMTP id 3f1490d57ef6-b99f0a0052fso6888020276.3 for ; Mon, 08 May 2023 15:00:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583223; x=1686175223; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=zvrzQZXuFLZVzyzSdbpkeKnq1Mi6dpkY47xXRlfYQiQ=; b=YLt74zwW0OQQBReGh4TRIHfOppLwHY50Ee0eFq8ibjo5jbgVvildNBJfKi1HjDJa4u Op3q9gJ/zjubHfcLj0M63FaRzFZoO3r0sBwu/zFnwboL22NRuuE3Z9XeCS+uSMT5+EuH fOlOBlNp7kzAiTDr1nDu4CvHVfRZLMk7T78j9QRQ6i7MiDUzvwPGQb5EfgsgOKHE41Ol tsMxLhWa8/M5Ks7MseYBm3UL3aGHyklaUzdQTYeu9bKic+GZVYiFNF6orm6kOJAa3Zmj szX9L7A/aW608XzoRUX/SY72cvdRiHd9nbNL/WWnbgn1OLjxNywkOgLk0Ivxj+Wyl+za mQ4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583223; x=1686175223; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=zvrzQZXuFLZVzyzSdbpkeKnq1Mi6dpkY47xXRlfYQiQ=; b=I+REBdiYR17uCHcgcuJXPk5qzvVEVmPZVy3irDTy1yhGxZClO+P2gSqGxxl+ZVJgqK K359a9E5xvYMjNloreoPOXnYB0pP816mIgEI6rTerCB+d1EtFrpixAw4WaBu25MTi2yP u05aCiBAtr/Fc5bx9+XNAbYN4Sn/2o19meTxYCKSaZGxvndOxH6J1iO8h0wmgUOvI3kM bJTR/cal45qSxMexm641qFYJIWHnV2TPEMHeQkt40UVON2kRKCSKlwJ1+CCMT+WOx9sk xoURpuJ9XJVTWfJkwHBjyBFft/K0BhAd2xfgCqf8JC5yFkIfl9tJmbYq6Bnd+t6u+3za Kf3Q== X-Gm-Message-State: AC+VfDzMrEzpnMy7sKFTg8fw1a7anv0EBsJd25WoLDRANE3LjJxptsai e6nuQQzgpkStwY+gP07ZhdLCAvh3lePl5RX4Ldluyg== X-Google-Smtp-Source: ACHHUZ4fVc9ejpl3aTRFg6oNztC1PIajZCeb5kc91yR0rbrtwbJZrqjUP4q6e/WWZluAoi8pj0/AmQ== X-Received: by 2002:a25:fc0a:0:b0:b8f:5639:cb8a with SMTP id v10-20020a25fc0a000000b00b8f5639cb8amr13722518ybd.9.1683583216870; Mon, 08 May 2023 15:00:16 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id d204-20020a251dd5000000b00b9a7db655ecsm2635062ybd.23.2023.05.08.15.00.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 15:00:16 -0700 (PDT) Date: Mon, 8 May 2023 18:00:15 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 11/15] revision.h: store hidden refs in a `strvec` Message-ID: <5b9814ad8c75029f836f051394ab387517ac3803.1683581621.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In subsequent commits, it will be convenient to have a 'const char **' of hidden refs (matching `transfer.hiderefs`, `uploadpack.hideRefs`, etc.), instead of a `string_list`. Convert spots throughout the tree that store the list of hidden refs from a `string_list` to a `strvec`. Note that in `parse_hide_refs_config()` there is an ugly const-cast used to avoid an extra copy of each value before trimming any trailing slash characters. This could instead be written as: ref = xstrdup(value); len = strlen(ref); while (len && ref[len - 1] == '/') ref[--len] = '\0'; strvec_push(hide_refs, ref); free(ref); but the double-copy (once when calling `xstrdup()`, and another via `strvec_push()`) is wasteful. Signed-off-by: Taylor Blau --- builtin/receive-pack.c | 4 ++-- ls-refs.c | 6 +++--- refs.c | 11 ++++++----- refs.h | 4 ++-- revision.c | 2 +- revision.h | 5 +++-- upload-pack.c | 10 +++++----- 7 files changed, 22 insertions(+), 20 deletions(-) diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index d22180435c..064df74715 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -89,7 +89,7 @@ static struct object_id push_cert_oid; static struct signature_check sigcheck; static const char *push_cert_nonce; static const char *cert_nonce_seed; -static struct string_list hidden_refs = STRING_LIST_INIT_DUP; +static struct strvec hidden_refs = STRVEC_INIT; static const char *NONCE_UNSOLICITED = "UNSOLICITED"; static const char *NONCE_BAD = "BAD"; @@ -2618,7 +2618,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix) packet_flush(1); oid_array_clear(&shallow); oid_array_clear(&ref); - string_list_clear(&hidden_refs, 0); + strvec_clear(&hidden_refs); free((void *)push_cert_nonce); return 0; } diff --git a/ls-refs.c b/ls-refs.c index c7ad39611a..d3d7e13e5a 100644 --- a/ls-refs.c +++ b/ls-refs.c @@ -71,7 +71,7 @@ struct ls_refs_data { unsigned symrefs; struct strvec prefixes; struct strbuf buf; - struct string_list hidden_refs; + struct strvec hidden_refs; unsigned unborn : 1; }; @@ -154,7 +154,7 @@ int ls_refs(struct repository *r, struct packet_reader *request) memset(&data, 0, sizeof(data)); strvec_init(&data.prefixes); strbuf_init(&data.buf, 0); - string_list_init_dup(&data.hidden_refs); + strvec_init(&data.hidden_refs); git_config(ls_refs_config, &data); @@ -196,7 +196,7 @@ int ls_refs(struct repository *r, struct packet_reader *request) packet_fflush(stdout); strvec_clear(&data.prefixes); strbuf_release(&data.buf); - string_list_clear(&data.hidden_refs, 0); + strvec_clear(&data.hidden_refs); return 0; } diff --git a/refs.c b/refs.c index 538bde644e..ec4d5b9101 100644 --- a/refs.c +++ b/refs.c @@ -1427,7 +1427,7 @@ char *shorten_unambiguous_ref(const char *refname, int strict) } int parse_hide_refs_config(const char *var, const char *value, const char *section, - struct string_list *hide_refs) + struct strvec *hide_refs) { const char *key; if (!strcmp("transfer.hiderefs", var) || @@ -1438,22 +1438,23 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti if (!value) return config_error_nonbool(var); - ref = xstrdup(value); + + /* drop const to remove trailing '/' characters */ + ref = (char *)strvec_push(hide_refs, value); len = strlen(ref); while (len && ref[len - 1] == '/') ref[--len] = '\0'; - string_list_append_nodup(hide_refs, ref); } return 0; } int ref_is_hidden(const char *refname, const char *refname_full, - const struct string_list *hide_refs) + const struct strvec *hide_refs) { int i; for (i = hide_refs->nr - 1; i >= 0; i--) { - const char *match = hide_refs->items[i].string; + const char *match = hide_refs->v[i]; const char *subject; int neg = 0; const char *p; diff --git a/refs.h b/refs.h index d672d636cf..a7751a1fc9 100644 --- a/refs.h +++ b/refs.h @@ -810,7 +810,7 @@ int update_ref(const char *msg, const char *refname, unsigned int flags, enum action_on_err onerr); int parse_hide_refs_config(const char *var, const char *value, const char *, - struct string_list *); + struct strvec *); /* * Check whether a ref is hidden. If no namespace is set, both the first and @@ -820,7 +820,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *, * the ref is outside that namespace, the first parameter is NULL. The second * parameter always points to the full ref name. */ -int ref_is_hidden(const char *, const char *, const struct string_list *); +int ref_is_hidden(const char *, const char *, const struct strvec *); /* Is this a per-worktree ref living in the refs/ namespace? */ int is_per_worktree_ref(const char *refname); diff --git a/revision.c b/revision.c index 89953592f9..7c9367a266 100644 --- a/revision.c +++ b/revision.c @@ -1558,7 +1558,7 @@ void init_ref_exclusions(struct ref_exclusions *exclusions) void clear_ref_exclusions(struct ref_exclusions *exclusions) { string_list_clear(&exclusions->excluded_refs, 0); - string_list_clear(&exclusions->hidden_refs, 0); + strvec_clear(&exclusions->hidden_refs); exclusions->hidden_refs_configured = 0; } diff --git a/revision.h b/revision.h index e8f6de9684..30b5b5919d 100644 --- a/revision.h +++ b/revision.h @@ -9,6 +9,7 @@ #include "commit-slab-decl.h" #include "ident.h" #include "list-objects-filter-options.h" +#include "strvec.h" /** * The revision walking API offers functions to build a list of revisions @@ -94,7 +95,7 @@ struct ref_exclusions { * Hidden refs is a list of patterns that is to be hidden via * `ref_is_hidden()`. */ - struct string_list hidden_refs; + struct strvec hidden_refs; /* * Indicates whether hidden refs have been configured. This is to @@ -109,7 +110,7 @@ struct ref_exclusions { */ #define REF_EXCLUSIONS_INIT { \ .excluded_refs = STRING_LIST_INIT_DUP, \ - .hidden_refs = STRING_LIST_INIT_DUP, \ + .hidden_refs = STRVEC_INIT, \ } struct oidset; diff --git a/upload-pack.c b/upload-pack.c index 08633dc121..d77d58bdde 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -69,7 +69,7 @@ struct upload_pack_data { struct object_array have_obj; struct oid_array haves; /* v2 only */ struct string_list wanted_refs; /* v2 only */ - struct string_list hidden_refs; + struct strvec hidden_refs; struct object_array shallows; struct string_list deepen_not; @@ -126,7 +126,7 @@ static void upload_pack_data_init(struct upload_pack_data *data) { struct string_list symref = STRING_LIST_INIT_DUP; struct string_list wanted_refs = STRING_LIST_INIT_DUP; - struct string_list hidden_refs = STRING_LIST_INIT_DUP; + struct strvec hidden_refs = STRVEC_INIT; struct object_array want_obj = OBJECT_ARRAY_INIT; struct object_array have_obj = OBJECT_ARRAY_INIT; struct oid_array haves = OID_ARRAY_INIT; @@ -161,7 +161,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data) { string_list_clear(&data->symref, 1); string_list_clear(&data->wanted_refs, 1); - string_list_clear(&data->hidden_refs, 0); + strvec_clear(&data->hidden_refs); object_array_clear(&data->want_obj); object_array_clear(&data->have_obj); oid_array_clear(&data->haves); @@ -1169,7 +1169,7 @@ static void receive_needs(struct upload_pack_data *data, /* return non-zero if the ref is hidden, otherwise 0 */ static int mark_our_ref(const char *refname, const char *refname_full, - const struct object_id *oid, const struct string_list *hidden_refs) + const struct object_id *oid, const struct strvec *hidden_refs) { struct object *o = lookup_unknown_object(the_repository, oid); @@ -1453,7 +1453,7 @@ static int parse_want(struct packet_writer *writer, const char *line, static int parse_want_ref(struct packet_writer *writer, const char *line, struct string_list *wanted_refs, - struct string_list *hidden_refs, + struct strvec *hidden_refs, struct object_array *want_obj) { const char *refname_nons; From patchwork Mon May 8 22:00:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96ECCC77B7F for ; Mon, 8 May 2023 22:00:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234561AbjEHWAo (ORCPT ); Mon, 8 May 2023 18:00:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234500AbjEHWA2 (ORCPT ); Mon, 8 May 2023 18:00:28 -0400 Received: from mail-yb1-xb34.google.com (mail-yb1-xb34.google.com [IPv6:2607:f8b0:4864:20::b34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A62549CE for ; Mon, 8 May 2023 15:00:22 -0700 (PDT) Received: by mail-yb1-xb34.google.com with SMTP id 3f1490d57ef6-b9a6ab9ede3so6672336276.2 for ; Mon, 08 May 2023 15:00:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583221; x=1686175221; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=rhIr3gh/rYbKoW0ZD837fYOPtntHhCQxbh/pYJvisp0=; b=4ukg/q+1Exf4Hg/imHsRDs+BtfECNo3AZzpkuqqU7lfqAS9FgqpBUUpg9rY2V3bc68 osilyxb/Y6Qpgr466e4CviAWsnpQCk+LCiwBsr+NwzliGg7RGtYCaXePWgs9W97fsL48 +c+zd689uC50zOquGBPhkH1D+2hL63k8d3jAJ7sxitTZwxfl7Mb/awGTGwgiz7sA06NM KH/oPKMSQONuPR4EI3MIGWAvSUNCh1UPIl8wbFM9gz6SFxH/zdkgP9GcD1OwhaATFIcM 0sZt2DmmDvziCTqNTgt3c06QFuZwBFjGbsqONbBBv4axv3xNz/aeWSAklEXlD9e987RU yCIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583221; x=1686175221; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=rhIr3gh/rYbKoW0ZD837fYOPtntHhCQxbh/pYJvisp0=; b=PEmmzuhN9zKQX0GPufd2bpIUW2P/e70xxkFTQ4npxKboqz65msDK+GnemW/Bqy5j72 nmMYVyHE+cGS+exdwaHaXE90JTILEk0GLDWNB/6AUaUz1H3Q6RHmnB1kavn0NEZlNpOs p9P0xY/ZZGswd3B2X3CcKm2IFS3AEcIcEMH0gGDPq25A9pzhFouCkAQgJw+IvfOYwx73 vk1d9CSla6rDGtD3IM4/QNVpU6V35KUXF0WO+9YExb7cM8kcxhP72Vat36XopLxLVsWA PnSzja5v8Q17nrznYlvu3thvkQNUqau5Eojd6R4UItenXlvr2A42+6NgcyVknxqVX60W kCqw== X-Gm-Message-State: AC+VfDw8z2pbNfQ5dOFvJ4hZdNo2W57Al3i/xkmK3MLW+0JWwK3o3n7F Pe8Ry9NItIRY1bWnfLv7/v64lZg/owiuG0ucNuIX+A== X-Google-Smtp-Source: ACHHUZ46STM1HRtJ2vPKS/HQiUs6sAxovX2ZBOOVfkFL76yb7wgqYH8rnF5em1txZ8LeCL/3EYsKKQ== X-Received: by 2002:a25:4f08:0:b0:b9d:bcb3:deae with SMTP id d8-20020a254f08000000b00b9dbcb3deaemr12066100ybb.45.1683583221255; Mon, 08 May 2023 15:00:21 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id p141-20020a257493000000b00b8c08669033sm2613497ybc.40.2023.05.08.15.00.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 15:00:20 -0700 (PDT) Date: Mon, 8 May 2023 18:00:19 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 12/15] refs/packed-backend.c: ignore complicated hidden refs rules Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In subsequent commits, we'll teach `receive-pack` and `upload-pack` to use the new skip-list feature in the packed-refs iterator by ignoring references which are mentioned via its respective hideRefs lists. However, the packed-ref skip lists cannot handle un-hiding rules (that begin with '!'), or namespace comparisons (that begin with '^'). Detect and avoid these cases by falling back to the normal enumeration without a skip list when such patterns exist. Signed-off-by: Taylor Blau --- refs/packed-backend.c | 19 +++++++++++++++++++ t/t1419-exclude-refs.sh | 10 ++++++++++ 2 files changed, 29 insertions(+) diff --git a/refs/packed-backend.c b/refs/packed-backend.c index ddfa9add14..7f09201f35 100644 --- a/refs/packed-backend.c +++ b/refs/packed-backend.c @@ -1013,6 +1013,25 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter, if (!excluded_patterns) return; + for (pattern = excluded_patterns; *pattern; pattern++) { + /* + * We also can't feed any excludes from hidden refs + * config sections, since later rules may override + * previous ones. For example, with rules "refs/foo" and + * "!refs/foo/bar", we should show "refs/foo/bar" (and + * everything underneath it), but the earlier exclusion + * would cause us to skip all of "refs/foo". We likewise + * don't implement the namespace stripping required for + * '^' rules. + * + * Both are possible to do, but complicated, so avoid + * populating the skip list at all if we see either of + * these patterns. + */ + if (**pattern == '!' || **pattern == '^') + return; + } + for (pattern = excluded_patterns; *pattern; pattern++) { struct skip_list_entry *e; diff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh index 051b5a54ce..026e4414cd 100755 --- a/t/t1419-exclude-refs.sh +++ b/t/t1419-exclude-refs.sh @@ -118,4 +118,14 @@ test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' ' assert_no_skips ' +test_expect_success 'for_each_ref__exclude(refs/heads/foo, !refs/heads/foo/1)' ' + # discards complex hidden ref rules + for_each_ref__exclude refs/heads refs/heads/foo "!refs/heads/foo/1" \ + >actual 2>perf && + for_each_ref >expect && + + test_cmp expect actual && + assert_no_skips +' + test_done From patchwork Mon May 8 22:00:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18776C7EE26 for ; Mon, 8 May 2023 22:00:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234395AbjEHWAz (ORCPT ); Mon, 8 May 2023 18:00:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234466AbjEHWAb (ORCPT ); Mon, 8 May 2023 18:00:31 -0400 Received: from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com [IPv6:2607:f8b0:4864:20::1134]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB773903E for ; Mon, 8 May 2023 15:00:25 -0700 (PDT) Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-55a20a56a01so91312117b3.3 for ; Mon, 08 May 2023 15:00:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583225; x=1686175225; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=aueWem9Nh6l2gkOjVe2yRUwgpcAYekX6zI5oPbxFPwM=; b=no9fTKDqCaD226BI8mbWcFfE3Cuo1sX+ryaTZmBLd6Tn3Wx014lKReTIAWkk0Ar/LI Wx3D5qYFa9JVaYJYTKmjAiIwdl90kM4USSHnLr2szMBOWYPohSzyc+gxei6wdMM+Tdim XyshWg9mXRjGiXMKEnzHP1TSG0li6LD1y2rTq5Pai+H1Mup2jgQKbICsl646krv+f1Hk /qJQV/bwWc8xMMheyTRPZXfo664gR4zziq/Ydb0QamFV3Lqj2JGYYNEcFqQYLCw9YyH4 F57101M23b0xL8TyMYp9Rkkvv+Enr3tMRDb2swvI4X1wuAh3ac/tUNhl0/t0KxvNQOs2 R36w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583225; x=1686175225; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=aueWem9Nh6l2gkOjVe2yRUwgpcAYekX6zI5oPbxFPwM=; b=Dw5FnXbzMlSzjjFyJPwGRTywj0emfQo0ZW/Etj3tA8gYFLCz6UTtDH+voVXGzgUwEZ aWZtCypy62CcxmmdL0h3LiXX03Imxjhs/DhL3/pi7DbzRzACwXkvFHWPd1xApWc8zv6G J8JTEAWsN/iCCwimdp55V3Hk+B4wFFnHo9ahtIRwfgqVV6h2CMnrJdgJw37097H4jxME 273TSu7nnYPAn+NueyoWTe2QBkK3GxwiGWVFtK3iFCa8mH2aXUjd7mn4CLYKVi747gSN FeyxEfNv8uBsafq2kagbOyvskc4gnqK3PO6uGgkjYzuaVWReQtiDD/hpHVbd0k5I37p2 PXmg== X-Gm-Message-State: AC+VfDwExTwLYgSGPDyKboXnzN0hEFSz/I5LUgblYAxFu8zrUki5ZLi+ wECyqS2VxODNovPoqLtLB0Us73RUfS7xGmlcRNiSYA== X-Google-Smtp-Source: ACHHUZ5QX3qRkdIzJ6F8/1SEWE3PF8kksHJBA+nWIoujnIEji53Ck6ZZeUKGJiCqANo+inMYLiXN8Q== X-Received: by 2002:a81:7d02:0:b0:55a:88f0:a49c with SMTP id y2-20020a817d02000000b0055a88f0a49cmr13911576ywc.33.1683583224734; Mon, 08 May 2023 15:00:24 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id d136-20020a0ddb8e000000b0054f6ca85641sm2809226ywe.99.2023.05.08.15.00.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 15:00:24 -0700 (PDT) Date: Mon, 8 May 2023 18:00:23 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 13/15] refs.h: let `for_each_namespaced_ref()` take excluded patterns Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The following commit will want to call `for_each_namespaced_ref()` with a list of excluded patterns. We could introduce a variant of that function, say, `for_each_namespaced_ref_exclude()` which takes the extra parameter, and reimplement the original function in terms of that. But all but one caller (in `http-backend.c`) will supply the new parameter, so add the new parameter to `for_each_namespaced_ref()` itself instead of introducing a new function. For now, supply NULL for the list of excluded patterns at all callers to avoid changing behavior, which we will do in the subsequent commit. Signed-off-by: Taylor Blau --- http-backend.c | 2 +- refs.c | 5 +++-- refs.h | 3 ++- upload-pack.c | 6 +++--- 4 files changed, 9 insertions(+), 7 deletions(-) diff --git a/http-backend.c b/http-backend.c index ac146d85c5..ad500683c8 100644 --- a/http-backend.c +++ b/http-backend.c @@ -559,7 +559,7 @@ static void get_info_refs(struct strbuf *hdr, char *arg UNUSED) } else { select_getanyfile(hdr); - for_each_namespaced_ref(show_text_ref, &buf); + for_each_namespaced_ref(NULL, show_text_ref, &buf); send_strbuf(hdr, "text/plain", &buf); } strbuf_release(&buf); diff --git a/refs.c b/refs.c index ec4d5b9101..95a7db9563 100644 --- a/refs.c +++ b/refs.c @@ -1660,13 +1660,14 @@ int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_dat DO_FOR_EACH_INCLUDE_BROKEN, cb_data); } -int for_each_namespaced_ref(each_ref_fn fn, void *cb_data) +int for_each_namespaced_ref(const char **exclude_patterns, + each_ref_fn fn, void *cb_data) { struct strbuf buf = STRBUF_INIT; int ret; strbuf_addf(&buf, "%srefs/", get_git_namespace()); ret = do_for_each_ref(get_main_ref_store(the_repository), - buf.buf, NULL, fn, 0, 0, cb_data); + buf.buf, exclude_patterns, fn, 0, 0, cb_data); strbuf_release(&buf); return ret; } diff --git a/refs.h b/refs.h index a7751a1fc9..f23626beca 100644 --- a/refs.h +++ b/refs.h @@ -372,7 +372,8 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern, const char *prefix, void *cb_data); int head_ref_namespaced(each_ref_fn fn, void *cb_data); -int for_each_namespaced_ref(each_ref_fn fn, void *cb_data); +int for_each_namespaced_ref(const char **exclude_patterns, + each_ref_fn fn, void *cb_data); /* can be used to learn about broken ref and symref */ int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data); diff --git a/upload-pack.c b/upload-pack.c index d77d58bdde..7c646ea5bd 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -854,7 +854,7 @@ static void deepen(struct upload_pack_data *data, int depth) * marked with OUR_REF. */ head_ref_namespaced(check_ref, data); - for_each_namespaced_ref(check_ref, data); + for_each_namespaced_ref(NULL, check_ref, data); get_reachable_list(data, &reachable_shallows); result = get_shallow_commits(&reachable_shallows, @@ -1378,7 +1378,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc, if (advertise_refs) data.no_done = 1; head_ref_namespaced(send_ref, &data); - for_each_namespaced_ref(send_ref, &data); + for_each_namespaced_ref(NULL, send_ref, &data); /* * fflush stdout before calling advertise_shallow_grafts because send_ref * uses stdio. @@ -1388,7 +1388,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc, packet_flush(1); } else { head_ref_namespaced(check_ref, &data); - for_each_namespaced_ref(check_ref, &data); + for_each_namespaced_ref(NULL, check_ref, &data); } if (!advertise_refs) { From patchwork Mon May 8 22:00:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4402C7EE22 for ; Mon, 8 May 2023 22:01:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234482AbjEHWBF (ORCPT ); Mon, 8 May 2023 18:01:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234531AbjEHWAg (ORCPT ); Mon, 8 May 2023 18:00:36 -0400 Received: from mail-yw1-x112f.google.com (mail-yw1-x112f.google.com [IPv6:2607:f8b0:4864:20::112f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 66B1186A5 for ; Mon, 8 May 2023 15:00:30 -0700 (PDT) Received: by mail-yw1-x112f.google.com with SMTP id 00721157ae682-55aa1da9d4aso91380377b3.2 for ; Mon, 08 May 2023 15:00:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583229; x=1686175229; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=nRKz2UuVP7e31x/bU+NO+yIpBZtjGBCmmgfAWDVba2E=; b=msOqXUCnC3c0wxUGjOmtnm758F/38tB/upeKdV87bhElO/U27B2Ziw6iIbNG6Zft5j LzghpilI4zzWics4cPqRH/PK9c9ISmMNjhgUVI1VCT6QitgOCk5idAe0WWTY2eQWHR14 oAmPLYhMrnzb4HVgEmwP3uIigBX3Q+BQhuMlROdsO4w6WQfVW+spJRQN6nP7HnIZGdC5 LlN4aGcN3X3F++13fpP9hAzmP3Hs2Kp1ANaSWTddljlJ4LFjw8pVYmltDK8Z0F/XI/DQ PDdnD8U+IOgMNyw6E+ItFnwm0trCiL1OW+81wdwsJvP8aw/gThlcQJnxWfSj1gUuE/PQ efZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583229; x=1686175229; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nRKz2UuVP7e31x/bU+NO+yIpBZtjGBCmmgfAWDVba2E=; b=bpf56F1elXoT8SvPmxhXbLIxRff1OBzUMV+U9HMSSSSGhO5C801OiTPM92wEvYGCdx aX8JydAYZzKrFmrCO9izAl+4Y4GARnS4J12GTviGAhOViQkFKm0EpVe9QDGvq6h3kUnh CUWoLaUlgkPquuunL6FxHY7J7RARAieujYC23G+FYnZfNOc6NfSegKxzIdTQWeYFv7Z7 V0W8biPgyHCE4V0EJm1hrtpy2JCGpMCIBUsect4OKBLSSFF4jyfDxXoCgwEHNhIWseaM g1t7bSLHix2v8IQ98XWyXFNrw6EvERbkM9JvYVh+SuYygF21LAVgCMKRcNiepNho98Zt Q8OA== X-Gm-Message-State: AC+VfDxTWEbURlTGgAB9CCWUgzzkvb7qR7ihNLg/1TY7GZkQVJWi9nd4 kgAOA22YMZITee0fVxRCiJec1L4DUpYvwg6PaUVXhQ== X-Google-Smtp-Source: ACHHUZ60Urd9Nz9EMsCfaTBAXY/a21BxUtw5rUHsnp8yNCSM9oKo4FoU5lDbeOKJ8Nq55cgEaQ0Ppg== X-Received: by 2002:a0d:ca15:0:b0:55a:8626:7725 with SMTP id m21-20020a0dca15000000b0055a86267725mr12437559ywd.26.1683583227991; Mon, 08 May 2023 15:00:27 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id s12-20020a81bf4c000000b00556aa81f615sm1963925ywk.68.2023.05.08.15.00.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 15:00:27 -0700 (PDT) Date: Mon, 8 May 2023 18:00:26 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 14/15] upload-pack.c: avoid enumerating hidden refs where possible Message-ID: <44bbf85e73676b2c89a82c09f7d355122ce6e805.1683581621.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In a similar fashion as a previous commit, teach `upload-pack` to avoid enumerating hidden references where possible. Note, however, that there are certain cases where cannot avoid enumerating even hidden references, in particular when either of: - `uploadpack.allowTipSHA1InWant`, or - `uploadpack.allowReachableSHA1InWant` are set, corresponding to `ALLOW_TIP_SHA1` and `ALLOW_REACHABLE_SHA1`, respectively. When either of these bits are set, upload-pack's `is_our_ref()` function needs to consider the `HIDDEN_REF` bit of the referent's object flags. So we must visit all references, including the hidden ones, in order to mark their referents with the `HIDDEN_REF` bit. When neither `ALLOW_TIP_SHA1` nor `ALLOW_REACHABLE_SHA1` are set, the `is_our_ref()` function considers only the `OUR_REF` bit, and not the `HIDDEN_REF` one. `OUR_REF` is applied via `mark_our_ref()`, and only to objects at the tips of non-hidden references, so we do not need to visit hidden references in this case. When neither of those bits are set, `upload-pack` can potentially avoid enumerating a large number of references. In the same example as a previous commit (linux.git with one hidden reference per commit, "refs/pull/N"): $ printf 0000 >in $ hyperfine --warmup=1 \ 'git -c transfer.hideRefs=refs/pull upload-pack . --- upload-pack.c | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-) diff --git a/upload-pack.c b/upload-pack.c index 7c646ea5bd..0162fffce0 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -601,11 +601,32 @@ static int get_common_commits(struct upload_pack_data *data, } } +static int allow_hidden_refs(enum allow_uor allow_uor) +{ + return allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1); +} + +static void for_each_namespaced_ref_1(each_ref_fn fn, + struct upload_pack_data *data) +{ + /* + * If `data->allow_uor` allows updating hidden refs, we need to + * mark all references (including hidden ones), to check in + * `is_our_ref()` below. + * + * Otherwise, we only care about whether each reference's object + * has the OUR_REF bit set or not, so do not need to visit + * hidden references. + */ + if (allow_hidden_refs(data->allow_uor)) + for_each_namespaced_ref(NULL, fn, data); + else + for_each_namespaced_ref(data->hidden_refs.v, fn, data); +} + static int is_our_ref(struct object *o, enum allow_uor allow_uor) { - int allow_hidden_ref = (allow_uor & - (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1)); - return o->flags & ((allow_hidden_ref ? HIDDEN_REF : 0) | OUR_REF); + return o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF); } /* @@ -854,7 +875,7 @@ static void deepen(struct upload_pack_data *data, int depth) * marked with OUR_REF. */ head_ref_namespaced(check_ref, data); - for_each_namespaced_ref(NULL, check_ref, data); + for_each_namespaced_ref_1(check_ref, data); get_reachable_list(data, &reachable_shallows); result = get_shallow_commits(&reachable_shallows, @@ -1378,7 +1399,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc, if (advertise_refs) data.no_done = 1; head_ref_namespaced(send_ref, &data); - for_each_namespaced_ref(NULL, send_ref, &data); + for_each_namespaced_ref_1(send_ref, &data); /* * fflush stdout before calling advertise_shallow_grafts because send_ref * uses stdio. @@ -1388,7 +1409,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc, packet_flush(1); } else { head_ref_namespaced(check_ref, &data); - for_each_namespaced_ref(NULL, check_ref, &data); + for_each_namespaced_ref_1(check_ref, &data); } if (!advertise_refs) { From patchwork Mon May 8 22:00:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13235184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B0E4C77B7F for ; Mon, 8 May 2023 22:01:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234556AbjEHWBI (ORCPT ); Mon, 8 May 2023 18:01:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234536AbjEHWAg (ORCPT ); Mon, 8 May 2023 18:00:36 -0400 Received: from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com [IPv6:2607:f8b0:4864:20::1134]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B7CC93CA for ; Mon, 8 May 2023 15:00:33 -0700 (PDT) Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-55a2691637bso76215937b3.0 for ; Mon, 08 May 2023 15:00:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20221208.gappssmtp.com; s=20221208; t=1683583232; x=1686175232; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=5e9AE+0n2v0CbPbkLsydiTTp1m/H7ePHit13Sy0o89g=; b=sS2HTpJCle+kc0cM/9VhXC3024muXNwTyEs3VM6AMU68xvX2LQ8/YKIYZrHC/r6znj r1iZAo1nGKTnRQIdKCa3d1YyTVUo/oWRlzBQW+BK5Fy8MvLNnlDtQEtb6oj3jECXJQXw j2/Gjz5EZm0IzAQvvESXhYQ8me3VQNI4yc0MihEKZB007yuU8uSEfSqRBjZ2ejjyqdbE 8les0rv7TKIJpgkwspo2Nm3sMAA0veTSyUVlDhgnVEYnOKAJ0ZDJP3Tthd8+nf3UNopI HPK9PkQTdxdyEE/rtH6sPIj7MIMB6JMtv9CLXx3kv4FX/gItf5Ndq6MvjfH/8RtsDkIv +xkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683583232; x=1686175232; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5e9AE+0n2v0CbPbkLsydiTTp1m/H7ePHit13Sy0o89g=; b=Qzhwmxn7RUb9V0ZKiHEDeVSp9ixkQobpc41jHdau7bGOzn7qP6G7srm/L1jgq31nH3 IRrVK7rH14hBnXZ4HRGyfFh2FiM6M+Qtrf9JViNuoTZMJ9QMbb/9culdlUodQXpPeB3w IzjrQ10USmgA5QHW5tKedJ1sV59xKGElf9popj5AXo5JiH83KmCdhGsjoggkraCNvjZ3 ldhBqgnJkkTz28gZCXxx1ap64KvQgzQOtV54INkbyucW5EmFPGt+Q5kP3KoebSTqv1Id sbmHBhpNJW1yyR650QAqy9uH7WFh/l2Hp/7Quoa4I8q2LtIqq2NqD2m4J+rYtoeWXefl KvWQ== X-Gm-Message-State: AC+VfDzaVSpwRcTPqNlvXbBlWWJveAR9uWu5FljkCBOKp5+FwX6hsnuf E7vxPQsjWqTaHwsu/K+DoBG6u96NDHxopXVBGW88DQ== X-Google-Smtp-Source: ACHHUZ7+FUBEjc+nBqf67+9BRDq6U/1Jr07fMBY4UuHgtaNhD4BWj0VfjXBLpwDKtbWZPAtSteoh3A== X-Received: by 2002:a81:6008:0:b0:54f:8b56:bb3 with SMTP id u8-20020a816008000000b0054f8b560bb3mr13101365ywb.5.1683583232111; Mon, 08 May 2023 15:00:32 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id l4-20020a81d544000000b0055b46dac76esm2792843ywj.116.2023.05.08.15.00.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 May 2023 15:00:31 -0700 (PDT) Date: Mon, 8 May 2023 18:00:30 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Derrick Stolee , Junio C Hamano Subject: [PATCH 15/15] builtin/receive-pack.c: avoid enumerating hidden references Message-ID: <7d3383083db80cab6ee022508c4ec0dbec698b26.1683581621.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Now that `refs_for_each_fullref_in()` has the ability to avoid enumerating references matching certain pattern(s), use that to avoid visiting hidden refs when constructing the ref advertisement via receive-pack. Note that since this exclusion is best-effort, we still need `show_ref_cb()` to check whether or not each reference is hidden or not before including it in the advertisement. As was the case when applying this same optimization to `upload-pack`, `receive-pack`'s reference advertisement phase can proceed much quicker by avoiding enumerating references that will not be part of the advertisement. (Below, we're still using linux.git with one hidden refs/pull/N ref per commit): $ hyperfine -L v ,.compile 'git{v} -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git' Benchmark 1: git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git Time (mean ± σ): 89.1 ms ± 1.7 ms [User: 82.0 ms, System: 7.0 ms] Range (min … max): 87.7 ms … 95.5 ms 31 runs Benchmark 2: git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git Time (mean ± σ): 4.5 ms ± 0.2 ms [User: 0.5 ms, System: 3.9 ms] Range (min … max): 4.1 ms … 5.6 ms 508 runs Summary 'git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git' ran 20.00 ± 1.05 times faster than 'git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git' Signed-off-by: Taylor Blau --- builtin/receive-pack.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index 064df74715..b954bcf802 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -336,7 +336,8 @@ static void write_head_info(void) { static struct oidset seen = OIDSET_INIT; - for_each_ref(show_ref_cb, &seen); + refs_for_each_fullref_in(get_main_ref_store(the_repository), "", + hidden_refs.v, show_ref_cb, &seen); for_each_alternate_ref(show_one_alternate_ref, &seen); oidset_clear(&seen); if (!sent_capabilities)