From patchwork Fri Jul 31 20:26:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11695433 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8B217138C for ; Fri, 31 Jul 2020 20:26:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 74AA622B3F for ; Fri, 31 Jul 2020 20:26:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="csvfID7Y" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727911AbgGaU0b (ORCPT ); Fri, 31 Jul 2020 16:26:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725767AbgGaU0b (ORCPT ); Fri, 31 Jul 2020 16:26:31 -0400 Received: from mail-qk1-x744.google.com (mail-qk1-x744.google.com [IPv6:2607:f8b0:4864:20::744]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B482C061574 for ; Fri, 31 Jul 2020 13:26:31 -0700 (PDT) Received: by mail-qk1-x744.google.com with SMTP id l64so23187553qkb.8 for ; Fri, 31 Jul 2020 13:26:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=PJrFU+75QPJCoYigEyOKR4oxa0fQ98NHJ7TU05rAv8E=; b=csvfID7YFSoANJcxeuMijGx1tw6c4kOjLaYxU5Fp61ZphCilLcwT9LoMQSL141B8/E hsiL9w8m0u38dSi9sBg8DibvYAqYE68Kpv+Sdw/CzwVNByVwm+f5dnZEKWLgpdxgJ4vv tUpsv/Eas3lvo53fHex8DZx90Dd3X1uaXOTrqbGNepJvUJYmkXGzbFVwCuLXD9e6+Wk4 NVh9ohRJj+/5DdL0ZvHLSEMSp7wvLLTQ6y8jX/m4zIJf45QWsVHMe6rEyl7aJ6SIVZ0E 5vQSOsM/V4MPQUg42ZhOuoKReaAN7+GZSIGeF2LaCLHKZ1V4EpjwPjlaqC80esRDkLeI dbsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=PJrFU+75QPJCoYigEyOKR4oxa0fQ98NHJ7TU05rAv8E=; b=fanffB5vm2Rr/qrroePMtJ/jEizDb7dzbejJhaAn8eG78TaxUXTKkIHrOmE1ET1WCs JeMPoSwkK2GS2yzh5UoVSfdQecFe+gABpZbYMwn/2j3sqK+VoJ/bB8Epwka24YiFJddx JUw9sa2PUn2myFTde+9HuQT8Frlv2b3QavmWPTZ+p9WGHFQQFo3l2VhxD6Suxfa6hhTx 63t7mPJ7I3tTNbJhxfhMIylzvgvDAp93slN3/M26p5Fm2ixMvmI4KHeHaprqo7wsz3E7 sg4gQxZDb0C7z4WstLZeEFoMJUW4RVHlQbIGfoJvqynNSYqICv1NFL6EDtSYOHRnzn+O MLDA== X-Gm-Message-State: AOAM530eN6E5RHCxR3Jhq7Ez6LBfoVJ9hHhbf1nhQt3fz/AfQq8yzWT9 bL055z+tCCRpfScVRAU5qKBUtm9ZhGapXA== X-Google-Smtp-Source: ABdhPJydTs3/6HFQ4pUN8iw3LsAe95utQMQmNnvH4u4FAGQImHKyXXqfFGQFWbTSNFwKL6FszfXzQA== X-Received: by 2002:a37:d201:: with SMTP id f1mr5771946qkj.188.1596227189014; Fri, 31 Jul 2020 13:26:29 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:1861:1305:b592:e057]) by smtp.gmail.com with ESMTPSA id d143sm9612151qkc.59.2020.07.31.13.26.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 31 Jul 2020 13:26:28 -0700 (PDT) Date: Fri, 31 Jul 2020 16:26:26 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, chriscool@tuxfamily.org, gitster@pobox.com, szeder.dev@gmail.com Subject: [PATCH v3 1/4] list_objects_filter_options: introduce 'list_object_filter_config_name' Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In a subsequent commit, we will add configuration options that are specific to each kind of object filter, in which case it is handy to have a function that translates between 'enum list_objects_filter_choice' and an appropriate configuration-friendly string. Signed-off-by: Taylor Blau --- list-objects-filter-options.c | 23 +++++++++++++++++++++++ list-objects-filter-options.h | 6 ++++++ 2 files changed, 29 insertions(+) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 3553ad7b0a..92b408c0c8 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -15,6 +15,29 @@ static int parse_combine_filter( const char *arg, struct strbuf *errbuf); +const char *list_object_filter_config_name(enum list_objects_filter_choice c) +{ + switch (c) { + case LOFC_DISABLED: + /* we have no name for "no filter at all" */ + break; + case LOFC_BLOB_NONE: + return "blob:none"; + case LOFC_BLOB_LIMIT: + return "blob:limit"; + case LOFC_TREE_DEPTH: + return "tree"; + case LOFC_SPARSE_OID: + return "sparse:oid"; + case LOFC_COMBINE: + return "combine"; + case LOFC__COUNT: + /* not a real filter type; just the count of all filters */ + break; + } + BUG("list_object_filter_choice_name: invalid argument '%d'", c); +} + /* * Parse value of the argument to the "filter" keyword. * On the command line this looks like: diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index 73fffa4ad7..01767c3c96 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -17,6 +17,12 @@ enum list_objects_filter_choice { LOFC__COUNT /* must be last */ }; +/* + * Returns a configuration key suitable for describing the given object filter, + * e.g.: "blob:none", "combine", etc. + */ +const char *list_object_filter_config_name(enum list_objects_filter_choice c); + struct list_objects_filter_options { /* * 'filter_spec' is the raw argument value given on the command line From patchwork Fri Jul 31 20:26:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11695435 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 696E4138C for ; Fri, 31 Jul 2020 20:26:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 44CDA22B42 for ; Fri, 31 Jul 2020 20:26:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="VBylh+n0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728722AbgGaU0g (ORCPT ); Fri, 31 Jul 2020 16:26:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725767AbgGaU0f (ORCPT ); Fri, 31 Jul 2020 16:26:35 -0400 Received: from mail-qt1-x842.google.com (mail-qt1-x842.google.com [IPv6:2607:f8b0:4864:20::842]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A31EC061574 for ; Fri, 31 Jul 2020 13:26:35 -0700 (PDT) Received: by mail-qt1-x842.google.com with SMTP id c12so14930600qtn.9 for ; Fri, 31 Jul 2020 13:26:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=9GWbJA9YASmBtONCnU4hMgTAsMYXTfXJYg/mVDMPmNE=; b=VBylh+n0h4SW79l+aNu26usBzAYSSh1pqsXgpIStVOz7zdxkLVQJA1B1o1doZV8dM0 UUDN8vEOeBYNTEK0x7hhWmMqh+sJVxrS3YJt4uG1RF0QbChtpWvInfUnS5f67l5jkTCF CRROoy8CqQtiO7Z0qlrxSyhEOPE6Tixu6k0yytzvZvyxq071Sw8bGi04Of6uBSyvVWSx kIEbReog3+E+ej3E9QRWSO26w+0rw1o0nGPfdLodUGnSIjomoV7b7iPaqH/mUesDI+5e nmbOo2SBcjWAbm3e9B0C202m13PXlhX2jYTlq/unkOmTncEHp5OcsBaYdGItwyUR+Q4n +O9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=9GWbJA9YASmBtONCnU4hMgTAsMYXTfXJYg/mVDMPmNE=; b=U5us+d+SwVZ1hq7EwkzCYPRjC4Zqsg1ahQXkjJUxBXmLULBb+9nZAJrCbOmvBZ92Eu zc2OPlwwxkK6B3sP+TByztbnhU+248XHZM4TIj6cwiFWmuTjqy8+t4Z4Lhu5PI65+lLG 6AMwRaaqLOVGcRXEDc75QKz12/PJeZP2Wf1DRGQ62A8kPHqsIGCxvzLvepRd15cgbn5C 7D/VkPuepnyT6vsryFwMSsgydHuxDLJEJ5nOP15YRqMcMuV/mDnzGrtgdMb1+HoK0FSQ LKzCWTcd9xYHOm1htMg5RdgSOfUcozLQaws3T1hEZnaKsfHQ5lmBACbAMD6k4kPkmNpa CNUg== X-Gm-Message-State: AOAM532eQhcHPy+FHjy9qZjxeIJA6l0zWMj8G83gh87gBshMoeFMCnI6 M6XrMWOoxY7dlkSxCve2jLRJTB92+ZyNXA== X-Google-Smtp-Source: ABdhPJwPTGa0OLiqeYPcri9hopqGz0NTzO/0ACTI3xNAsDoaLO5hGzSHZw4LNIpm8N556s+K+nxIlA== X-Received: by 2002:ac8:4652:: with SMTP id f18mr5399977qto.142.1596227193951; Fri, 31 Jul 2020 13:26:33 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:1861:1305:b592:e057]) by smtp.gmail.com with ESMTPSA id n23sm9042609qkk.105.2020.07.31.13.26.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 31 Jul 2020 13:26:32 -0700 (PDT) Date: Fri, 31 Jul 2020 16:26:31 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, chriscool@tuxfamily.org, gitster@pobox.com, szeder.dev@gmail.com Subject: [PATCH v3 2/4] upload-pack.c: allow banning certain object filter(s) Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Git clients may ask the server for a partial set of objects, where the set of objects being requested is refined by one or more object filters. Server administrators can configure 'git upload-pack' to allow or ban these filters by setting the 'uploadpack.allowFilter' variable to 'true' or 'false', respectively. However, administrators using bitmaps may wish to allow certain kinds of object filters, but ban others. Specifically, they may wish to allow object filters that can be optimized by the use of bitmaps, while rejecting other object filters which aren't and represent a perceived performance degradation (as well as an increased load factor on the server). Allow configuring 'git upload-pack' to support object filters on a case-by-case basis by introducing two new configuration variables: - 'uploadpackfilter.allow' - 'uploadpackfilter..allow' where '' may be one of 'blobNone', 'blobLimit', 'tree', and so on. Setting the second configuration variable for any valid value of '' explicitly allows or disallows restricting that kind of object filter. If a client requests the object filter and the respective configuration value is not set, 'git upload-pack' will default to the value of 'uploadpackfilter.allow', which itself defaults to 'true' to maintain backwards compatibility. Note that this differs from 'uploadpack.allowfilter', which controls whether or not the 'filter' capability is advertised. Signed-off-by: Taylor Blau --- Documentation/config/uploadpack.txt | 12 +++++ t/t5616-partial-clone.sh | 24 +++++++++ upload-pack.c | 80 +++++++++++++++++++++++++++++ 3 files changed, 116 insertions(+) diff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt index ed1c835695..fffe8ac648 100644 --- a/Documentation/config/uploadpack.txt +++ b/Documentation/config/uploadpack.txt @@ -57,6 +57,18 @@ uploadpack.allowFilter:: If this option is set, `upload-pack` will support partial clone and partial fetch object filtering. +uploadpackfilter.allow:: + Provides a default value for unspecified object filters (see: the + below configuration variable). + Defaults to `true`. + +uploadpackfilter..allow:: + Explicitly allow or ban the object filter corresponding to + ``, where `` may be one of: `blob:none`, + `blob:limit`, `tree`, `sparse:oid`, or `combine`. If using + combined filters, both `combine` and all of the nested filter + kinds must be allowed. Defaults to `uploadpackfilter.allow`. + uploadpack.allowRefInWant:: If this option is set, `upload-pack` will support the `ref-in-want` feature of the protocol version 2 `fetch` command. This feature diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 37de0afb02..0d46b5a2f8 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -235,6 +235,30 @@ test_expect_success 'implicitly construct combine: filter with repeated flags' ' test_cmp unique_types.expected unique_types.actual ' +test_expect_success 'upload-pack fails banned object filters' ' + test_config -C srv.bare uploadpackfilter.blob:none.allow false && + test_must_fail ok=sigpipe git clone --no-checkout --filter=blob:none \ + "file://$(pwd)/srv.bare" pc3 2>err && + test_i18ngrep "filter '\''blob:none'\'' not supported" err +' + +test_expect_success 'upload-pack fails banned combine object filters' ' + test_config -C srv.bare uploadpackfilter.allow false && + test_config -C srv.bare uploadpackfilter.combine.allow true && + test_config -C srv.bare uploadpackfilter.tree.allow true && + test_config -C srv.bare uploadpackfilter.blob:none.allow false && + test_must_fail ok=sigpipe git clone --no-checkout --filter=tree:1 \ + --filter=blob:none "file://$(pwd)/srv.bare" pc3 2>err && + test_i18ngrep "filter '\''blob:none'\'' not supported" err +' + +test_expect_success 'upload-pack fails banned object filters with fallback' ' + test_config -C srv.bare uploadpackfilter.allow false && + test_must_fail ok=sigpipe git clone --no-checkout --filter=blob:none \ + "file://$(pwd)/srv.bare" pc3 2>err && + test_i18ngrep "filter '\''blob:none'\'' not supported" err +' + test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' ' rm -rf src dst && git init src && diff --git a/upload-pack.c b/upload-pack.c index 8673741070..ed2098edd0 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -88,6 +88,7 @@ struct upload_pack_data { enum allow_uor allow_uor; struct list_objects_filter_options filter_options; + struct string_list allowed_filters; struct packet_writer writer; @@ -103,6 +104,7 @@ struct upload_pack_data { unsigned no_progress : 1; unsigned use_include_tag : 1; unsigned allow_filter : 1; + unsigned allow_filter_fallback : 1; unsigned done : 1; /* v2 only */ unsigned allow_ref_in_want : 1; /* v2 only */ @@ -120,6 +122,7 @@ static void upload_pack_data_init(struct upload_pack_data *data) struct string_list deepen_not = STRING_LIST_INIT_DUP; struct string_list uri_protocols = STRING_LIST_INIT_DUP; struct object_array extra_edge_obj = OBJECT_ARRAY_INIT; + struct string_list allowed_filters = STRING_LIST_INIT_DUP; memset(data, 0, sizeof(*data)); data->symref = symref; @@ -131,6 +134,8 @@ static void upload_pack_data_init(struct upload_pack_data *data) data->deepen_not = deepen_not; data->uri_protocols = uri_protocols; data->extra_edge_obj = extra_edge_obj; + data->allowed_filters = allowed_filters; + data->allow_filter_fallback = 1; packet_writer_init(&data->writer, 1); data->keepalive = 5; @@ -147,6 +152,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data) string_list_clear(&data->deepen_not, 0); object_array_clear(&data->extra_edge_obj); list_objects_filter_release(&data->filter_options); + string_list_clear(&data->allowed_filters, 1); free((char *)data->pack_objects_hook); } @@ -984,6 +990,50 @@ static int process_deepen_not(const char *line, struct string_list *deepen_not, return 0; } +static int allows_filter_choice(struct upload_pack_data *data, + enum list_objects_filter_choice c) +{ + const char *key = list_object_filter_config_name(c); + struct string_list_item *item = string_list_lookup(&data->allowed_filters, + key); + if (item) + return (intptr_t) item->util; + return data->allow_filter_fallback; +} + +static struct list_objects_filter_options *banned_filter( + struct upload_pack_data *data, + struct list_objects_filter_options *opts) +{ + size_t i; + + if (!allows_filter_choice(data, opts->choice)) + return opts; + + if (opts->choice == LOFC_COMBINE) + for (i = 0; i < opts->sub_nr; i++) { + struct list_objects_filter_options *sub = &opts->sub[i]; + if (banned_filter(data, sub)) + return sub; + } + return NULL; +} + +static void die_if_using_banned_filter(struct upload_pack_data *data) +{ + struct list_objects_filter_options *banned = banned_filter(data, + &data->filter_options); + struct strbuf buf = STRBUF_INIT; + if (!banned) + return; + + strbuf_addf(&buf, "git upload-pack: filter '%s' not supported", + list_object_filter_config_name(banned->choice)); + + packet_writer_error(&data->writer, "%s\n", buf.buf); + die("%s", buf.buf); +} + static void receive_needs(struct upload_pack_data *data, struct packet_reader *reader) { @@ -1014,6 +1064,7 @@ static void receive_needs(struct upload_pack_data *data, die("git upload-pack: filtering capability not negotiated"); list_objects_filter_die_if_populated(&data->filter_options); parse_list_objects_filter(&data->filter_options, arg); + die_if_using_banned_filter(data); continue; } @@ -1171,6 +1222,32 @@ static int find_symref(const char *refname, const struct object_id *oid, return 0; } +static int parse_object_filter_config(const char *var, const char *value, + struct upload_pack_data *data) +{ + struct strbuf buf = STRBUF_INIT; + const char *sub, *key; + size_t sub_len; + + if (parse_config_key(var, "uploadpackfilter", &sub, &sub_len, &key)) + return 0; + + if (!sub) { + if (!strcmp(key, "allow")) + data->allow_filter_fallback = git_config_bool(var, value); + return 0; + } + + strbuf_add(&buf, sub, sub_len); + + if (!strcmp(key, "allow")) + string_list_insert(&data->allowed_filters, buf.buf)->util = + (void *)(intptr_t)git_config_bool(var, value); + + strbuf_release(&buf); + return 0; +} + static int upload_pack_config(const char *var, const char *value, void *cb_data) { struct upload_pack_data *data = cb_data; @@ -1210,6 +1287,8 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data) return git_config_string(&data->pack_objects_hook, var, value); } + parse_object_filter_config(var, value, data); + return parse_hide_refs_config(var, value, "uploadpack"); } @@ -1390,6 +1469,7 @@ static void process_args(struct packet_reader *request, if (data->allow_filter && skip_prefix(arg, "filter ", &p)) { list_objects_filter_die_if_populated(&data->filter_options); parse_list_objects_filter(&data->filter_options, p); + die_if_using_banned_filter(data); continue; } From patchwork Fri Jul 31 20:26:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11695437 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E9D9F138C for ; Fri, 31 Jul 2020 20:26:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D34BE22B43 for ; Fri, 31 Jul 2020 20:26:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="xtMz92yk" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728910AbgGaU0k (ORCPT ); Fri, 31 Jul 2020 16:26:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725767AbgGaU0j (ORCPT ); Fri, 31 Jul 2020 16:26:39 -0400 Received: from mail-qt1-x842.google.com (mail-qt1-x842.google.com [IPv6:2607:f8b0:4864:20::842]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3FF69C061574 for ; Fri, 31 Jul 2020 13:26:39 -0700 (PDT) Received: by mail-qt1-x842.google.com with SMTP id 6so23983735qtt.0 for ; Fri, 31 Jul 2020 13:26:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=d0ZQ34NpC6OH8RzVlQ0NYyoQ5i5b0HhWQ9vPQ1ESBhw=; b=xtMz92ykSCNrD2i4PG21kuZ0fpqkyXH5U1b+sOlNwlPuuMBprgz3JgrEmB3NgonG0B 59pbLYQ+DRTL+KCEa3gHhVnEzcP6nfeHbzANcY59GIOeRilkCX/fFvnC6CDXG5UL75vH k8i9zE/llfVc/08lCVsbGDJQRjLFB3k3P6OwnvkiwFkJ+gIBDMYl3c0xgc2tgQutq28Q hXrH/vfqtRt1rrDmw+2y2i8hUWNrrTUZ3vWpP4bqsPqP3Bi7/5nMsiN8Upkb81CQRjFY i4nHAdjG5dnJnS6jxXseaz/lhFFSbpQb2B9NwCX6tMpIu7q6ozVa3iWnhbBaQQCw82Gl yTrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=d0ZQ34NpC6OH8RzVlQ0NYyoQ5i5b0HhWQ9vPQ1ESBhw=; b=k4NNTSXuxcnevYlvhkn5l/8qlDNDMIooK22A5KXCVpMGSldY2Ft7M7/Qu+pPva3cSd MGO1TETkU9Je9AsDIh+PUomM5iTM4c9ij6xwVKh0X/FUTbkyOy+I2uWzywuR60wOs9HC w1QjMCVRfKTnobrtqi1KdNSMzBEBZHxg0P/YEs/d4wD/8gTXhKAQFms2KEcbTdpG0ffl uAkULvnv+Cc2dzhV7tbNXSruorLrEsCi/Kuq2UpAYmqEN7hEZjg3x08AT8LEF2pmTiJ3 SzYDTFfn/+35p1lLtzjkNjGzxfGms6wJt8fylLYDtzQackTzBIwGGJILaRR54yeNsXwe PO0Q== X-Gm-Message-State: AOAM532hKQOUI5/XoMeI0e6TZt9OZNjh0LYrGA7+AGBzSB8QN5PF7DRX 3innF6y4pkSqtz9M3ICyMgZqhr4U+/nIEQ== X-Google-Smtp-Source: ABdhPJz3oRcn3z3tv0zWeSgAb1POnVYYk9KDqtaY55nc54OV2xPLYiRmTpY7x/YxUaoiN1UCM93Xcg== X-Received: by 2002:ac8:47cb:: with SMTP id d11mr3688142qtr.311.1596227197761; Fri, 31 Jul 2020 13:26:37 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:1861:1305:b592:e057]) by smtp.gmail.com with ESMTPSA id d46sm10655786qtk.37.2020.07.31.13.26.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 31 Jul 2020 13:26:36 -0700 (PDT) Date: Fri, 31 Jul 2020 16:26:35 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, chriscool@tuxfamily.org, gitster@pobox.com, szeder.dev@gmail.com Subject: [PATCH v3 3/4] upload-pack.c: pass 'struct list_objects_filter_options *' Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The 'allows_filter_choice' function used to take an 'enum list_objects_filter_choice', but in a future commit it will be more convenient for it to accept the whole 'struct list_objects_filter_options', for e.g., to inspect the value of '->tree_exclude_depth'. Signed-off-by: Taylor Blau --- upload-pack.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/upload-pack.c b/upload-pack.c index ed2098edd0..5fa22da31f 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -991,9 +991,9 @@ static int process_deepen_not(const char *line, struct string_list *deepen_not, } static int allows_filter_choice(struct upload_pack_data *data, - enum list_objects_filter_choice c) + struct list_objects_filter_options *opts) { - const char *key = list_object_filter_config_name(c); + const char *key = list_object_filter_config_name(opts->choice); struct string_list_item *item = string_list_lookup(&data->allowed_filters, key); if (item) @@ -1007,7 +1007,7 @@ static struct list_objects_filter_options *banned_filter( { size_t i; - if (!allows_filter_choice(data, opts->choice)) + if (!allows_filter_choice(data, opts)) return opts; if (opts->choice == LOFC_COMBINE) From patchwork Fri Jul 31 20:26:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11695439 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1C22213B1 for ; Fri, 31 Jul 2020 20:26:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F2EBD22B3F for ; Fri, 31 Jul 2020 20:26:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="MUO29w6B" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729051AbgGaU0o (ORCPT ); Fri, 31 Jul 2020 16:26:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49110 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725767AbgGaU0n (ORCPT ); Fri, 31 Jul 2020 16:26:43 -0400 Received: from mail-qk1-x743.google.com (mail-qk1-x743.google.com [IPv6:2607:f8b0:4864:20::743]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5440C061574 for ; Fri, 31 Jul 2020 13:26:43 -0700 (PDT) Received: by mail-qk1-x743.google.com with SMTP id l23so30046124qkk.0 for ; Fri, 31 Jul 2020 13:26:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=bRPN9/aHbdi2iL3c6SJIcY26jU7g2txXhiSCHDyZDZ0=; b=MUO29w6BPrJlN0Bv/bj19PIkEZiIo0AARUmpYD3zB0sTJFi3U28WtfaxjXVDRy+Isr thb4MEKJ5I/eCo7zLBMwMWE33eeEAWRxxiNovmVOKKbf2JCv3VQjTaSdggHEITLzVSwQ JfS33MYYpxOQAe4dZkA9wXjnpT70qR7cjU8PAIgxY/goSG7pMKehk18H7TthGoH3njd9 WgRthLtjuFEBfOXARZXP7yaDIW3f5Il3gd7Xw7mcfEUfIawf+IA5lPL3RkXwyqWgMxfp WyG+pwwX8KDLgmhqbug4loUlkjFivC/O8SukfvHpRF+V7KEtpHVfcKFmJpFtlCLCR8Wp +rQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=bRPN9/aHbdi2iL3c6SJIcY26jU7g2txXhiSCHDyZDZ0=; b=GwC29XYeWyOSZ8j4ry8CWTx228HvjHuHgt+SqD1YJ7ogSZk7SOhXJ1QDfIMcHnTMBx 8bYcQUG20mFITuuB/eLpUxI4/FLgtV3T6U5IgUEBC3hf9pcs8Fquwn9Q4XWI8jW8cwZg KI4I78pkOX0OhAP3RvJ5uHS01ask5jDhqC0LNmK2cNhTOl1fesCnoUn9hpRL6BHUIXt3 tM9UqM4UAKIRKF7GWTnw4iM+ObZ980S0Q5C8pmSJCaGerWbHmomoViUdrHMvgWcHTxoy uoAY3RmdThmUqXFtWPtIj1dT7Ob12ccvQ1S3EtKYWSxOSx9R3t+X0SRgPQgH3IQdHD+e eC6A== X-Gm-Message-State: AOAM532RvtYMaf7TPAHej9m5+2kYB50hClRI5Kufzi/ihUylDfuYCdXY +nuid7y/7Wxdza1lk8H9OhqfUjl1v5i/Sw== X-Google-Smtp-Source: ABdhPJyK0tDeadu7TcbpqhT4NgwhDZ7gZ9aXXHAEicF1W1eDKt4pZ7iIhSVDZogzC879N+VgP/84nw== X-Received: by 2002:a05:620a:122c:: with SMTP id v12mr5920899qkj.113.1596227202098; Fri, 31 Jul 2020 13:26:42 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:1861:1305:b592:e057]) by smtp.gmail.com with ESMTPSA id k48sm2903348qtk.44.2020.07.31.13.26.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 31 Jul 2020 13:26:40 -0700 (PDT) Date: Fri, 31 Jul 2020 16:26:39 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, chriscool@tuxfamily.org, gitster@pobox.com, szeder.dev@gmail.com Subject: [PATCH v3 4/4] upload-pack.c: introduce 'uploadpackfilter.tree.maxDepth' Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In b79cf959b2 (upload-pack.c: allow banning certain object filter(s), 2020-02-26), we introduced functionality to disallow certain object filters from being chosen from within 'git upload-pack'. Traditionally, administrators use this functionality to disallow filters that are known to perform slowly, for e.g., those that do not have bitmap-level filtering. In the past, the '--filter=tree:' was one such filter that does not have bitmap-level filtering support, and so was likely to be banned by administrators. However, in the previous couple of commits, we introduced bitmap-level filtering for the case when 'n' is equal to '0', i.e., as if we had a '--filter=tree:none' choice. While it would be sufficient to simply write $ git config uploadpackfilter.tree.allow true (since it would allow all values of 'n'), we would like to be able to allow this filter for certain values of 'n', i.e., those no greater than some pre-specified maximum. In order to do this, introduce a new configuration key, as follows: $ git config uploadpackfilter.tree.maxDepth where '' specifies the maximum allowed value of 'n' in the filter 'tree:n'. Administrators who wish to allow for only the value '0' can write: $ git config uploadpackfilter.tree.allow true $ git config uploadpackfilter.tree.maxDepth 0 which allows '--filter=tree:0', but no other values. Unfortunately, since the tree depth is an unsigned long, we can't use, say, -1 as a sentinel value, and so we must also keep track of "have we set this" as well as "to what value". Signed-off-by: Taylor Blau --- Documentation/config/uploadpack.txt | 6 ++++++ t/t5616-partial-clone.sh | 9 +++++++++ upload-pack.c | 27 ++++++++++++++++++++++++++- 3 files changed, 41 insertions(+), 1 deletion(-) diff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt index fffe8ac648..ee7b3ac94f 100644 --- a/Documentation/config/uploadpack.txt +++ b/Documentation/config/uploadpack.txt @@ -69,6 +69,12 @@ uploadpackfilter..allow:: combined filters, both `combine` and all of the nested filter kinds must be allowed. Defaults to `uploadpackfilter.allow`. +uploadpackfilter.tree.maxDepth:: + Only allow `--filter=tree=` when `n` is no more than the value of + `uploadpackfilter.tree.maxDepth`. If set, this also implies + `uploadpackfilter.tree.allow=true`, unless this configuration + variable had already been set. Has no effect if unset. + uploadpack.allowRefInWant:: If this option is set, `upload-pack` will support the `ref-in-want` feature of the protocol version 2 `fetch` command. This feature diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 0d46b5a2f8..35cb6a34a3 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -259,6 +259,15 @@ test_expect_success 'upload-pack fails banned object filters with fallback' ' test_i18ngrep "filter '\''blob:none'\'' not supported" err ' +test_expect_success 'upload-pack limits tree depth filters' ' + test_config -C srv.bare uploadpackfilter.allow false && + test_config -C srv.bare uploadpackfilter.tree.allow true && + test_config -C srv.bare uploadpackfilter.tree.maxDepth 0 && + test_must_fail ok=sigpipe git clone --no-checkout --filter=tree:1 \ + "file://$(pwd)/srv.bare" pc3 2>err && + test_i18ngrep "filter '\''tree'\'' not supported (maximum depth: 0, but got: 1)" err +' + test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' ' rm -rf src dst && git init src && diff --git a/upload-pack.c b/upload-pack.c index 5fa22da31f..131445b212 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -105,6 +105,7 @@ struct upload_pack_data { unsigned use_include_tag : 1; unsigned allow_filter : 1; unsigned allow_filter_fallback : 1; + unsigned long tree_filter_max_depth; unsigned done : 1; /* v2 only */ unsigned allow_ref_in_want : 1; /* v2 only */ @@ -136,6 +137,7 @@ static void upload_pack_data_init(struct upload_pack_data *data) data->extra_edge_obj = extra_edge_obj; data->allowed_filters = allowed_filters; data->allow_filter_fallback = 1; + data->tree_filter_max_depth = ULONG_MAX; packet_writer_init(&data->writer, 1); data->keepalive = 5; @@ -996,8 +998,17 @@ static int allows_filter_choice(struct upload_pack_data *data, const char *key = list_object_filter_config_name(opts->choice); struct string_list_item *item = string_list_lookup(&data->allowed_filters, key); + int allowed = -1; if (item) - return (intptr_t) item->util; + allowed = (intptr_t) item->util; + + if (allowed != 0 && + opts->choice == LOFC_TREE_DEPTH && + opts->tree_exclude_depth > data->tree_filter_max_depth) + return 0; + + if (allowed > -1) + return allowed; return data->allow_filter_fallback; } @@ -1029,6 +1040,11 @@ static void die_if_using_banned_filter(struct upload_pack_data *data) strbuf_addf(&buf, "git upload-pack: filter '%s' not supported", list_object_filter_config_name(banned->choice)); + if (banned->choice == LOFC_TREE_DEPTH && + data->tree_filter_max_depth != ULONG_MAX) + strbuf_addf(&buf, _(" (maximum depth: %lu, but got: %lu)"), + data->tree_filter_max_depth, + banned->tree_exclude_depth); packet_writer_error(&data->writer, "%s\n", buf.buf); die("%s", buf.buf); @@ -1243,6 +1259,15 @@ static int parse_object_filter_config(const char *var, const char *value, if (!strcmp(key, "allow")) string_list_insert(&data->allowed_filters, buf.buf)->util = (void *)(intptr_t)git_config_bool(var, value); + else if (!strcmp(buf.buf, "tree") && !strcmp(key, "maxdepth")) { + if (!value) { + strbuf_release(&buf); + return config_error_nonbool(var); + } + string_list_insert(&data->allowed_filters, buf.buf)->util = + (void *)(intptr_t)1; + data->tree_filter_max_depth = git_config_ulong(var, value); + } strbuf_release(&buf); return 0;