From patchwork Thu Jul 2 20:06:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11640361 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 30DED161F for ; Thu, 2 Jul 2020 20:06:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1AC0D208B8 for ; Thu, 2 Jul 2020 20:06:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="tI3J6WEF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726288AbgGBUGZ (ORCPT ); Thu, 2 Jul 2020 16:06:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43104 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725915AbgGBUGY (ORCPT ); Thu, 2 Jul 2020 16:06:24 -0400 Received: from mail-qv1-xf44.google.com (mail-qv1-xf44.google.com [IPv6:2607:f8b0:4864:20::f44]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30274C08C5C1 for ; Thu, 2 Jul 2020 13:06:24 -0700 (PDT) Received: by mail-qv1-xf44.google.com with SMTP id d12so13316242qvn.0 for ; Thu, 02 Jul 2020 13:06:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=VHjQQvjLvnTP6bgCUi8/1pjYXinQAj7ArgIl/GxoQbs=; b=tI3J6WEFN2uMip5pCVGeJaXurs0fHLFX0bCXkCn2fFaHPzsMq+3sZBNa9F0Ug28deK ojY+JuGWspuFn4Eo8hj+LoAWC5IuWK1rpMhZLmzME0Rjb+lWY46T7dqEN9p2gGZe5hEf n+ifllRooFXmCiQVlI7El+xJJvINRliTqiBJ+uRRhZgI7oB12wliA52Ha0MMgr06Z22W nvHTio+UTEshJ0EDeoRU75Q4E/5NKzWL3cZ8ZPQQJ4LHCZW4wz+6nwdVvUXM4zLzZRCL tHHBCfPZWDkBAfzTAyzmV1YNA20yzFFEklHLjLmn+KukbP4rMDKtRjmmwFBBRzGoZ3Ij 23Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=VHjQQvjLvnTP6bgCUi8/1pjYXinQAj7ArgIl/GxoQbs=; b=DzULYU9BP9Pf6fqf7fppN3MpzM6YGh3RQsCP5IeEhfqXJy4TMXHUmB5jB6S11JBd+9 EOG9J6l1pKeg+WAi3p5vAoJ/EqV4pmg4N7dH/etUQbiY5p5HaaUuSH6LHXoxGdNZwNWi yLljZcaHe4NRTgbvUT+ZE8fZ2mM8RZ2KTGwP2pfIyl7etjuWi4ABCPUf0qDbGDgcS7hb +y/NncPtjLFnSgWu/3DgXWVxMn910wsDNdrqP0/u1jAbCSCMSYOzpjDa4vSqFP0xLW8o gIibBNdCmIx1OTs7WuiVhPcLcvqlWOJ56KSkgM+CTDE9AlO+gsy5Ud/16Oukl7mJ2yhL wjFw== X-Gm-Message-State: AOAM531AypNrZytk1e8B7oIVsB80szZUsKdprSlUNJqfcuaD3z6x6C4c khCeAW0g659BB82x+EyJSrr28A9ccAibOA== X-Google-Smtp-Source: ABdhPJyFTcXiyVwN9eX2LoLUvPVuxT/mf2ttyJoBo7Mhp/QAJjD28WV1GY6GaPI1oAfo6olYmZd1Lw== X-Received: by 2002:a0c:e554:: with SMTP id n20mr31534442qvm.14.1593720382898; Thu, 02 Jul 2020 13:06:22 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:650d:8c1a:48a0:61b2]) by smtp.gmail.com with ESMTPSA id q28sm9962548qtk.13.2020.07.02.13.06.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Jul 2020 13:06:22 -0700 (PDT) Date: Thu, 2 Jul 2020 16:06:21 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, chriscool@tuxfamily.org Subject: [PATCH 1/4] list_objects_filter_options: introduce 'list_object_filter_config_name' Message-ID: <738d97be4645435c333111a06560926a701a5bea.1593720075.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In a subsequent commit, we will add configuration options that are specific to each kind of object filter, in which case it is handy to have a function that translates between 'enum list_objects_filter_choice' and an appropriate configuration-friendly string. Signed-off-by: Taylor Blau --- list-objects-filter-options.c | 23 +++++++++++++++++++++++ list-objects-filter-options.h | 6 ++++++ 2 files changed, 29 insertions(+) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 3553ad7b0a..92b408c0c8 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -15,6 +15,29 @@ static int parse_combine_filter( const char *arg, struct strbuf *errbuf); +const char *list_object_filter_config_name(enum list_objects_filter_choice c) +{ + switch (c) { + case LOFC_DISABLED: + /* we have no name for "no filter at all" */ + break; + case LOFC_BLOB_NONE: + return "blob:none"; + case LOFC_BLOB_LIMIT: + return "blob:limit"; + case LOFC_TREE_DEPTH: + return "tree"; + case LOFC_SPARSE_OID: + return "sparse:oid"; + case LOFC_COMBINE: + return "combine"; + case LOFC__COUNT: + /* not a real filter type; just the count of all filters */ + break; + } + BUG("list_object_filter_choice_name: invalid argument '%d'", c); +} + /* * Parse value of the argument to the "filter" keyword. * On the command line this looks like: diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index 73fffa4ad7..01767c3c96 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -17,6 +17,12 @@ enum list_objects_filter_choice { LOFC__COUNT /* must be last */ }; +/* + * Returns a configuration key suitable for describing the given object filter, + * e.g.: "blob:none", "combine", etc. + */ +const char *list_object_filter_config_name(enum list_objects_filter_choice c); + struct list_objects_filter_options { /* * 'filter_spec' is the raw argument value given on the command line From patchwork Thu Jul 2 20:06:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11640363 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9602C13BD for ; Thu, 2 Jul 2020 20:06:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 710DD20890 for ; Thu, 2 Jul 2020 20:06:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="Mmc4sKMf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726290AbgGBUGg (ORCPT ); Thu, 2 Jul 2020 16:06:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43136 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725915AbgGBUGg (ORCPT ); Thu, 2 Jul 2020 16:06:36 -0400 Received: from mail-qv1-xf44.google.com (mail-qv1-xf44.google.com [IPv6:2607:f8b0:4864:20::f44]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0CD59C08C5C1 for ; Thu, 2 Jul 2020 13:06:36 -0700 (PDT) Received: by mail-qv1-xf44.google.com with SMTP id m8so9080789qvk.7 for ; Thu, 02 Jul 2020 13:06:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=BHm+QVQOdE70LXEkxMdBPbTQ/qU7ng3hqD4zm4GJbHQ=; b=Mmc4sKMfvnVedka2/TxvGnZUuVMMhQwUipnVjUgMQrfzG5XE2DMwrb9aPGfSckV9Cd ZTyLzCctYX8akA29p5e3RVAVt2Ddi8A6JgOD9vZ003RodU1r3lgaU1MYnvjUQrNfCdiX 9bVQuQu5KNGQEUkKi9sYM0NQ0hdH5mhqeVtOZ/QcfY1kO8xq7F9xIUtiMGqPgj/wA0E1 dI1v0xOKF7ICqq+etE/cL3kTr8k3FWIe3cZL8jFZ8aTfjaYXrfZf6KlS+5whJ9Wht5Gs nlyw/fXLNn5UErb+WxC9Tx4JrwYW1TjzXUffIQrQfQ5Dya61EU/6MbApP3MJBr0+4gy/ 5IZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=BHm+QVQOdE70LXEkxMdBPbTQ/qU7ng3hqD4zm4GJbHQ=; b=OyaOqDqhO+Z9N4XklgIw92ZFU5ZbUPaLfmneh6RK77SpMNy8ZOEU3oNvC2WB1zFgkK Kc+Ga7v9/EgHRxxQ0QuhXjy1uTlcmczyIF7oJeq0UARVGtpyJ4QQjQDI9EiilhgA5R51 wp6oKYRVOq/U/LO7eF7KqrjB9HSGkC64LB52aASspEZWOys+3aa7GuC51ygx/6LamwGW DHqP6ZsS2Rp1sjet1dNWwsRJ6k3C2f1rWdyppXYBtVoV4sS6JauctTF0rbIu7gRV/VuX H8XoTNyLzJvX/oNTxy4r79BpbXz6qaKhUsU/xQ9qSZLfAOoPcIJQep1bJKVnbfoJkRBp etxA== X-Gm-Message-State: AOAM533wcLIJvQpmwkPsvT27oTAWqonHK7b8RDsFn/GxqeoTSXdqHbOw LpVA15RN35mJnCLrezVp4tlwSUEKhaHOAw== X-Google-Smtp-Source: ABdhPJyFZ9PVw1xcNnF0ZVKEhsgKbO8uJEbxHgHR62TIAI6spTQMi33rTZX11H86rf6eVq7z53o+jA== X-Received: by 2002:a05:6214:1882:: with SMTP id cx2mr32533167qvb.240.1593720394681; Thu, 02 Jul 2020 13:06:34 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:650d:8c1a:48a0:61b2]) by smtp.gmail.com with ESMTPSA id w204sm9442065qka.41.2020.07.02.13.06.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Jul 2020 13:06:34 -0700 (PDT) Date: Thu, 2 Jul 2020 16:06:32 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, chriscool@tuxfamily.org Subject: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s) Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Git clients may ask the server for a partial set of objects, where the set of objects being requested is refined by one or more object filters. Server administrators can configure 'git upload-pack' to allow or ban these filters by setting the 'uploadpack.allowFilter' variable to 'true' or 'false', respectively. However, administrators using bitmaps may wish to allow certain kinds of object filters, but ban others. Specifically, they may wish to allow object filters that can be optimized by the use of bitmaps, while rejecting other object filters which aren't and represent a perceived performance degradation (as well as an increased load factor on the server). Allow configuring 'git upload-pack' to support object filters on a case-by-case basis by introducing two new configuration variables: - 'uploadpack.filter.allow' - 'uploadpack.filter..allow' where '' may be one of 'blobNone', 'blobLimit', 'tree', and so on. Setting the second configuration variable for any valid value of '' explicitly allows or disallows restricting that kind of object filter. If a client requests the object filter and the respective configuration value is not set, 'git upload-pack' will default to the value of 'uploadpack.filter.allow', which itself defaults to 'true' to maintain backwards compatibility. Note that this differs from 'uploadpack.allowfilter', which controls whether or not the 'filter' capability is advertised. Signed-off-by: Taylor Blau --- Documentation/config/uploadpack.txt | 16 ++++++ t/t5616-partial-clone.sh | 26 ++++++++++ upload-pack.c | 78 +++++++++++++++++++++++++++++ 3 files changed, 120 insertions(+) diff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt index ed1c835695..fd4970306c 100644 --- a/Documentation/config/uploadpack.txt +++ b/Documentation/config/uploadpack.txt @@ -57,6 +57,22 @@ uploadpack.allowFilter:: If this option is set, `upload-pack` will support partial clone and partial fetch object filtering. +uploadpack.filter.allow:: + Provides a default value for unspecified object filters (see: the + below configuration variable). + Defaults to `true`. + +uploadpack.filter..allow:: + Explicitly allow or ban the object filter corresponding to + ``, where `` may be one of: `blob:none`, + `blob:limit`, `tree`, `sparse:oid`, or `combine`. If using + combined filters, both `combine` and all of the nested filter + kinds must be allowed. Defaults to `uploadpack.filter.allow`. ++ +Note that the dot between 'filter' and '' is both non-standard +and intentional. This is done to avoid a parsing ambiguity when +specifying this configuration as an argument to Git's top-level `-c`. + uploadpack.allowRefInWant:: If this option is set, `upload-pack` will support the `ref-in-want` feature of the protocol version 2 `fetch` command. This feature diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 8a27452a51..5dcd0b5656 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -235,6 +235,32 @@ test_expect_success 'implicitly construct combine: filter with repeated flags' ' test_cmp unique_types.expected unique_types.actual ' +test_expect_success 'upload-pack fails banned object filters' ' + # Test case-insensitivity by intentional use of "blob:None" rather than + # "blob:none". + test_config -C srv.bare uploadpack.filter.blob:None.allow false && + test_must_fail git clone --no-checkout --filter=blob:none \ + "file://$(pwd)/srv.bare" pc3 2>err && + test_i18ngrep "filter '\''blob:none'\'' not supported" err +' + +test_expect_success 'upload-pack fails banned combine object filters' ' + test_config -C srv.bare uploadpack.filter.allow false && + test_config -C srv.bare uploadpack.filter.combine.allow true && + test_config -C srv.bare uploadpack.filter.tree.allow true && + test_config -C srv.bare uploadpack.filter.blob:none.allow false && + test_must_fail git clone --no-checkout --filter=tree:1 \ + --filter=blob:none "file://$(pwd)/srv.bare" pc3 2>err && + test_i18ngrep "filter '\''blob:none'\'' not supported" err +' + +test_expect_success 'upload-pack fails banned object filters with fallback' ' + test_config -C srv.bare uploadpack.filter.allow false && + test_must_fail git clone --no-checkout --filter=blob:none \ + "file://$(pwd)/srv.bare" pc3 2>err && + test_i18ngrep "filter '\''blob:none'\'' not supported" err +' + test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' ' rm -rf src dst && git init src && diff --git a/upload-pack.c b/upload-pack.c index 39d0cf00be..a5f56d73cc 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -88,6 +88,7 @@ struct upload_pack_data { enum allow_uor allow_uor; struct list_objects_filter_options filter_options; + struct string_list allowed_filters; struct packet_writer writer; @@ -103,6 +104,7 @@ struct upload_pack_data { unsigned no_progress : 1; unsigned use_include_tag : 1; unsigned allow_filter : 1; + unsigned allow_filter_fallback : 1; unsigned done : 1; /* v2 only */ unsigned allow_ref_in_want : 1; /* v2 only */ @@ -120,6 +122,7 @@ static void upload_pack_data_init(struct upload_pack_data *data) struct string_list deepen_not = STRING_LIST_INIT_DUP; struct string_list uri_protocols = STRING_LIST_INIT_DUP; struct object_array extra_edge_obj = OBJECT_ARRAY_INIT; + struct string_list allowed_filters = STRING_LIST_INIT_DUP; memset(data, 0, sizeof(*data)); data->symref = symref; @@ -131,6 +134,8 @@ static void upload_pack_data_init(struct upload_pack_data *data) data->deepen_not = deepen_not; data->uri_protocols = uri_protocols; data->extra_edge_obj = extra_edge_obj; + data->allowed_filters = allowed_filters; + data->allow_filter_fallback = 1; packet_writer_init(&data->writer, 1); data->keepalive = 5; @@ -147,6 +152,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data) string_list_clear(&data->deepen_not, 0); object_array_clear(&data->extra_edge_obj); list_objects_filter_release(&data->filter_options); + string_list_clear(&data->allowed_filters, 1); free((char *)data->pack_objects_hook); } @@ -983,6 +989,47 @@ static int process_deepen_not(const char *line, struct string_list *deepen_not, return 0; } +static int allows_filter_choice(struct upload_pack_data *data, + enum list_objects_filter_choice c) +{ + const char *key = list_object_filter_config_name(c); + struct string_list_item *item = string_list_lookup(&data->allowed_filters, + key); + if (item) + return (intptr_t) item->util; + return data->allow_filter_fallback; +} + +static struct list_objects_filter_options *banned_filter( + struct upload_pack_data *data, + struct list_objects_filter_options *opts) +{ + size_t i; + + if (!allows_filter_choice(data, opts->choice)) + return opts; + + if (opts->choice == LOFC_COMBINE) + for (i = 0; i < opts->sub_nr; i++) { + struct list_objects_filter_options *sub = &opts->sub[i]; + if (banned_filter(data, sub)) + return sub; + } + return NULL; +} + +static void die_if_using_banned_filter(struct upload_pack_data *data) +{ + struct list_objects_filter_options *banned = banned_filter(data, + &data->filter_options); + if (!banned) + return; + + packet_writer_error(&data->writer, _("filter '%s' not supported\n"), + list_object_filter_config_name(banned->choice)); + die(_("git upload-pack: banned object filter requested")); +} + static void receive_needs(struct upload_pack_data *data, struct packet_reader *reader) { @@ -1013,6 +1060,7 @@ static void receive_needs(struct upload_pack_data *data, die("git upload-pack: filtering capability not negotiated"); list_objects_filter_die_if_populated(&data->filter_options); parse_list_objects_filter(&data->filter_options, arg); + die_if_using_banned_filter(data); continue; } @@ -1169,6 +1217,33 @@ static int find_symref(const char *refname, const struct object_id *oid, return 0; } +static void parse_object_filter_config(const char *var, const char *value, + struct upload_pack_data *data) +{ + struct strbuf spec = STRBUF_INIT; + const char *sub, *key; + size_t sub_len; + + if (parse_config_key(var, "uploadpack", &sub, &sub_len, &key)) + return; + if (!sub || !skip_prefix(sub, "filter.", &sub)) + return; + + if (sub != key) + strbuf_add(&spec, sub, key - sub - 1); + strbuf_tolower(&spec); + + if (!strcmp(key, "allow")) { + if (spec.len) + string_list_insert(&data->allowed_filters, spec.buf)->util = + (void *)(intptr_t)git_config_bool(var, value); + else + data->allow_filter_fallback = git_config_bool(var, value); + } + + strbuf_release(&spec); +} + static int upload_pack_config(const char *var, const char *value, void *cb_data) { struct upload_pack_data *data = cb_data; @@ -1208,6 +1283,8 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data) return git_config_string(&data->pack_objects_hook, var, value); } + parse_object_filter_config(var, value, data); + return parse_hide_refs_config(var, value, "uploadpack"); } @@ -1388,6 +1465,7 @@ static void process_args(struct packet_reader *request, if (data->allow_filter && skip_prefix(arg, "filter ", &p)) { list_objects_filter_die_if_populated(&data->filter_options); parse_list_objects_filter(&data->filter_options, p); + die_if_using_banned_filter(data); continue; } From patchwork Thu Jul 2 20:06:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11640365 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 36DE7174A for ; Thu, 2 Jul 2020 20:06:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1F97320890 for ; Thu, 2 Jul 2020 20:06:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="Dn0nM0bW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726039AbgGBUGk (ORCPT ); Thu, 2 Jul 2020 16:06:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43144 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725915AbgGBUGj (ORCPT ); Thu, 2 Jul 2020 16:06:39 -0400 Received: from mail-qt1-x844.google.com (mail-qt1-x844.google.com [IPv6:2607:f8b0:4864:20::844]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B94ADC08C5C1 for ; Thu, 2 Jul 2020 13:06:39 -0700 (PDT) Received: by mail-qt1-x844.google.com with SMTP id w34so2226679qte.1 for ; Thu, 02 Jul 2020 13:06:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=mEuDYfuT89zXW/cF6FPZo3KObUGTW86veuhPVxLmtjc=; b=Dn0nM0bWkh5M+o3vIorBX0Rb0/gcJuY7mL4kaXeCd97eibbOpNgSe1jw7UcWn5nhcC K2tFRWEVnT+LE8C3u5LixOzFx93XL5Hq3ifjtrQFvE4TL3aYR7ccZKxUF0sSkaGuJlCm 67E+24KZmGkekO1oxsGu4AMM8defQsJADfptQmcm4/IpOWCFq/0B1HFtVH7YE+PWSvIV 494wZb4hKBBSm1Y+PIMKx8TIDJHtUhcvyyDZD23Y508F+EWGJFUjwWEUt8mPOpaIS501 /nUJViU1HS2O1KkyNqVxpjZSbRZYNf887RSOU5g6pTcV5EgY5E6dsx64nyPhOP/eJXu2 Qe1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=mEuDYfuT89zXW/cF6FPZo3KObUGTW86veuhPVxLmtjc=; b=iNjapDvCd8fBkru5CpzsFgvLIYzEMqQR2BIEI2tuB0rJT4PEh/uz4WVK6mUURRdRCB sE5u2noyJd8vrE26ja7XV0CPZBwV/hVJ7YEROTJ2wt9n1q4eaZTHDiWs0w+FIEH5FNmS S3JiEqe9u/zPCokDdUox/Pnz+1SU2jVx82TOj+lujK1MX5PmKvqgHVSsFQRhkae4TSRl HhkdHsyd1ouwNOLj1xVJkCc8ugISuOoOoQlEl333jIpmOCYFr8iSvG/qYgAa9649E+gc eFdH8Pw9shcDNiLHyxOhiDNVlvKQrALp/vj9Oa4BQn/gwANUA7t4lyjju8YWOxlI/x2q 2CGw== X-Gm-Message-State: AOAM532uOqscNmaGuHE0euh4RYfR/ycURRXL0x9qIimZWqcJlf9kmDbn OG4OIL6nSaD8hJNEdYfmAaMb5Zz6oKBBfQ== X-Google-Smtp-Source: ABdhPJzbBWUnVGDHlREEjOaxLZqJuQ2cPZcKjBOvQxn052yVlSiyR0ZEeLdaX1BfX4FjzAletO+Oyg== X-Received: by 2002:ac8:168d:: with SMTP id r13mr33139237qtj.207.1593720398621; Thu, 02 Jul 2020 13:06:38 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:650d:8c1a:48a0:61b2]) by smtp.gmail.com with ESMTPSA id r188sm8799951qkf.128.2020.07.02.13.06.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Jul 2020 13:06:38 -0700 (PDT) Date: Thu, 2 Jul 2020 16:06:36 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, chriscool@tuxfamily.org Subject: [PATCH 3/4] upload-pack.c: pass 'struct list_objects_filter_options *' Message-ID: <3434bd5979499899e24528c9a578288f1c8d2669.1593720075.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The 'allows_filter_choice' function used to take an 'enum list_objects_filter_choice', but in a future commit it will be more convenient for it to accept the whole 'struct list_objects_filter_options', for e.g., to inspect the value of '->tree_exclude_depth'. Signed-off-by: Taylor Blau --- upload-pack.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/upload-pack.c b/upload-pack.c index a5f56d73cc..a014ae23a9 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -990,9 +990,9 @@ static int process_deepen_not(const char *line, struct string_list *deepen_not, } static int allows_filter_choice(struct upload_pack_data *data, - enum list_objects_filter_choice c) + struct list_objects_filter_options *opts) { - const char *key = list_object_filter_config_name(c); + const char *key = list_object_filter_config_name(opts->choice); struct string_list_item *item = string_list_lookup(&data->allowed_filters, key); if (item) @@ -1006,7 +1006,7 @@ static struct list_objects_filter_options *banned_filter( { size_t i; - if (!allows_filter_choice(data, opts->choice)) + if (!allows_filter_choice(data, opts)) return opts; if (opts->choice == LOFC_COMBINE) From patchwork Thu Jul 2 20:06:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11640367 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DD33A161F for ; Thu, 2 Jul 2020 20:06:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BD5B22088E for ; Thu, 2 Jul 2020 20:06:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="xAA/sh9f" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726119AbgGBUGn (ORCPT ); Thu, 2 Jul 2020 16:06:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725915AbgGBUGn (ORCPT ); Thu, 2 Jul 2020 16:06:43 -0400 Received: from mail-qk1-x741.google.com (mail-qk1-x741.google.com [IPv6:2607:f8b0:4864:20::741]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4902CC08C5C1 for ; Thu, 2 Jul 2020 13:06:43 -0700 (PDT) Received: by mail-qk1-x741.google.com with SMTP id e13so26872655qkg.5 for ; Thu, 02 Jul 2020 13:06:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=r/mDULSywTj1gnI63NMMQbObygwrg4AQY9hOWJYRbA4=; b=xAA/sh9foT0qmuc1iJcxXDyG9LrgqrSkjC3g0RL/YX6UdwWi74KzHv8ZGQlP5/lsvq pcS/Kq0siShS7VBXdzgk7YDkrMtPEKrKV+7vX92brnfx5Rjk6KIp84QusUZbNcR5T9dA vhIHcvwGKOPI2jkDR2pvYOWuklvMQEmd3lQ8aEfTbnIgz93OmuJfgaSc5UUE+ksTw7Dd gTgthO9gdWiuWpWgLa9+QvC+j4G1Ff8gm9JVAt5kRu/dOKE29SdT3Qda9SArEf5N92Qk Rrp8mcXeKPnIo1czyFlJ3b8LFHTv2MpqNV6tx+F591aJ+1nnOkZorixh+8Xysq6l4YpX 2LSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=r/mDULSywTj1gnI63NMMQbObygwrg4AQY9hOWJYRbA4=; b=oESt5WAe+oZ0jpHM7Z7L2qDdkpNnWV+m9nC1fnzNkSmRk7LzFXtIfyAKl1Ehik/PsM yjRlXsoy2l+S4KyuTw4D2RtIefCb+ErdK59NHZ85HTEag3DMMOKrpF7EA0iCZ4b9Kz/o Au8nHGaVAHnopsc83ZIVOlHVGbCcN/3av8i53AR/Of7tBDBAuZxp5nnkQ+Pjm3ijYB7I TcFlCYCmPlCNOf8G828J6ErZAEsCgVLdUvs8tlqGSpR5ra4IxMJmSnVdRA3QoNvmCd0k 6+3x7QVh7Xi5A+1b/OWKgSPGonbDRPEQU++ZybQ6INEwE+sNWIucyQU3YQqD6ffmooik U4Cw== X-Gm-Message-State: AOAM5336+c6JOytbusY+0QYNPeWKzfZGc+9cIlP3RNAKsdijJuALCkAU pcnqKDfev1ataHCEq+iGCCgLlJm6ZxgNgQ== X-Google-Smtp-Source: ABdhPJw4jmct3VBljiamK4P0a0ZIkTd9uzZcpEz2am33seWC0IsEAo+ohzXo5BdStkGpxyx8c6cNTQ== X-Received: by 2002:a37:470a:: with SMTP id u10mr33328022qka.202.1593720402089; Thu, 02 Jul 2020 13:06:42 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:650d:8c1a:48a0:61b2]) by smtp.gmail.com with ESMTPSA id j7sm9268646qtd.53.2020.07.02.13.06.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Jul 2020 13:06:41 -0700 (PDT) Date: Thu, 2 Jul 2020 16:06:40 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, chriscool@tuxfamily.org Subject: [PATCH 4/4] upload-pack.c: introduce 'uploadpack.filter.tree.maxDepth' Message-ID: <9fa765a71d25ef3462ce81cca9754daa9b2579b6.1593720075.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In b79cf959b2 (upload-pack.c: allow banning certain object filter(s), 2020-02-26), we introduced functionality to disallow certain object filters from being chosen from within 'git upload-pack'. Traditionally, administrators use this functionality to disallow filters that are known to perform slowly, for e.g., those that do not have bitmap-level filtering. In the past, the '--filter=tree:' was one such filter that does not have bitmap-level filtering support, and so was likely to be banned by administrators. However, in the previous couple of commits, we introduced bitmap-level filtering for the case when 'n' is equal to '0', i.e., as if we had a '--filter=tree:none' choice. While it would be sufficient to simply write $ git config uploadpack.filter.tree.allow true (since it would allow all values of 'n'), we would like to be able to allow this filter for certain values of 'n', i.e., those no greater than some pre-specified maximum. In order to do this, introduce a new configuration key, as follows: $ git config uploadpack.filter.tree.maxDepth where '' specifies the maximum allowed value of 'n' in the filter 'tree:n'. Administrators who wish to allow for only the value '0' can write: $ git config uploadpack.filter.tree.allow true $ git config uploadpack.filter.tree.maxDepth 0 which allows '--filter=tree:0', but no other values. Unfortunately, since the tree depth is an unsigned long, we can't use, say, -1 as a sentinel value, and so we must also keep track of "have we set this" as well as "to what value". Signed-off-by: Taylor Blau --- Documentation/config/uploadpack.txt | 6 ++++++ t/t5616-partial-clone.sh | 8 ++++++++ upload-pack.c | 32 ++++++++++++++++++++++++++--- 3 files changed, 43 insertions(+), 3 deletions(-) diff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt index fd4970306c..3671b62e4c 100644 --- a/Documentation/config/uploadpack.txt +++ b/Documentation/config/uploadpack.txt @@ -73,6 +73,12 @@ Note that the dot between 'filter' and '' is both non-standard and intentional. This is done to avoid a parsing ambiguity when specifying this configuration as an argument to Git's top-level `-c`. +uploadpack.filter.tree.maxDepth:: + Only allow `--filter=tree=` when `n` is no more than the value of + `uploadpack.filter.tree.maxDepth`. If set, this also implies + `uploadpack.filter.tree.allow=true`, unless this configuration + variable had already been set. Has no effect if unset. + uploadpack.allowRefInWant:: If this option is set, `upload-pack` will support the `ref-in-want` feature of the protocol version 2 `fetch` command. This feature diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 5dcd0b5656..8781a24cfe 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -261,6 +261,14 @@ test_expect_success 'upload-pack fails banned object filters with fallback' ' test_i18ngrep "filter '\''blob:none'\'' not supported" err ' +test_expect_success 'upload-pack limits tree depth filters' ' + test_config -C srv.bare uploadpack.filter.allow false && + test_config -C srv.bare uploadpack.filter.tree.allow true && + test_config -C srv.bare uploadpack.filter.tree.maxDepth 0 && + test_must_fail ok=sigpipe git clone --no-checkout --filter=tree:1 \ + "file://$(pwd)/srv.bare" pc3 +' + test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' ' rm -rf src dst && git init src && diff --git a/upload-pack.c b/upload-pack.c index a014ae23a9..8db1745b86 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -105,6 +105,7 @@ struct upload_pack_data { unsigned use_include_tag : 1; unsigned allow_filter : 1; unsigned allow_filter_fallback : 1; + unsigned long tree_filter_max_depth; unsigned done : 1; /* v2 only */ unsigned allow_ref_in_want : 1; /* v2 only */ @@ -136,6 +137,7 @@ static void upload_pack_data_init(struct upload_pack_data *data) data->extra_edge_obj = extra_edge_obj; data->allowed_filters = allowed_filters; data->allow_filter_fallback = 1; + data->tree_filter_max_depth = ULONG_MAX; packet_writer_init(&data->writer, 1); data->keepalive = 5; @@ -995,8 +997,17 @@ static int allows_filter_choice(struct upload_pack_data *data, const char *key = list_object_filter_config_name(opts->choice); struct string_list_item *item = string_list_lookup(&data->allowed_filters, key); + int allowed = -1; if (item) - return (intptr_t) item->util; + allowed = (intptr_t) item->util; + + if (allowed != 0 && + opts->choice == LOFC_TREE_DEPTH && + opts->tree_exclude_depth > data->tree_filter_max_depth) + return 0; + + if (allowed > -1) + return allowed; return data->allow_filter_fallback; } @@ -1022,11 +1033,22 @@ static void die_if_using_banned_filter(struct upload_pack_data *data) { struct list_objects_filter_options *banned = banned_filter(data, &data->filter_options); + struct strbuf buf = STRBUF_INIT; if (!banned) return; - packet_writer_error(&data->writer, _("filter '%s' not supported\n"), - list_object_filter_config_name(banned->choice)); + strbuf_addf(&buf, _("filter '%s' not supported"), + list_object_filter_config_name(banned->choice)); + if (banned->choice == LOFC_TREE_DEPTH && + data->tree_filter_max_depth != ULONG_MAX) + strbuf_addf(&buf, _(" (maximum depth: %lu, but got: %lu)"), + data->tree_filter_max_depth, + banned->tree_exclude_depth); + + packet_writer_error(&data->writer, "%s\n", buf.buf); + + strbuf_release(&buf); + die(_("git upload-pack: banned object filter requested")); } @@ -1239,6 +1261,10 @@ static void parse_object_filter_config(const char *var, const char *value, (void *)(intptr_t)git_config_bool(var, value); else data->allow_filter_fallback = git_config_bool(var, value); + } else if (!strcmp(spec.buf, "tree") && !strcmp(key, "maxdepth")) { + string_list_insert(&data->allowed_filters, "tree")->util + = (void *) (intptr_t) 1; + data->tree_filter_max_depth = git_config_ulong(var, value); } strbuf_release(&spec);