From patchwork Thu Jul 23 01:48:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11679521 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C61AF6C1 for ; Thu, 23 Jul 2020 01:49:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ADF5920781 for ; Thu, 23 Jul 2020 01:49:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="TCrBKwKe" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387447AbgGWBtC (ORCPT ); Wed, 22 Jul 2020 21:49:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729198AbgGWBtC (ORCPT ); Wed, 22 Jul 2020 21:49:02 -0400 Received: from mail-qk1-x741.google.com (mail-qk1-x741.google.com [IPv6:2607:f8b0:4864:20::741]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E574C0619DC for ; Wed, 22 Jul 2020 18:49:02 -0700 (PDT) Received: by mail-qk1-x741.google.com with SMTP id z15so3975496qki.10 for ; Wed, 22 Jul 2020 18:49:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=PJrFU+75QPJCoYigEyOKR4oxa0fQ98NHJ7TU05rAv8E=; b=TCrBKwKemG35vAzpMGn965tpORxTwgvZr8ogX+ggxEntFkPc5UVPScOYARRyViIUTn A7+zIkOMmnw5byGRq4m3dAcNYTiomx6TeEfm6j1ucWjZgfLYvodswbbUO7k+UbLDuvMM +gRerCEngVBwoAxYbazo8oR9mAe6pMi6sm6tlSO7dBM9xbQk0GZRJk2wNQRlbC2ySHhd V1NS9Kar7NEy6FzHCjgGEFbQ9PolT2hhS3CNEZ/GioBHdZotMrc64nffEYO0NsdpDU2f edlldPb9gjpfIl9O/Be+0Xewjm3MoxDTUQvZ1mmgF18H0+tWJ9rpU1A/kAWcfTp5PHiC jz/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=PJrFU+75QPJCoYigEyOKR4oxa0fQ98NHJ7TU05rAv8E=; b=kRH2jw7nD52LJ4XH6sNlZYzjQIz3IqTkqU60c46TMKuKIzSxBvf8sZhUhbLW3k71Ha HCfXPnfmdeAvbSzF85384o9FdC9oBJUu7xOFgc+xM06bG/WBoJOS2qdcJU/wORzcMfBg sawN1tjbAESkxHx0YsrHwR2EA4bC3qUawYn+akBgUUBsqcbISOSbBIZUBWkT/s2mhhnx WOB8hHFCZd2E7WGmMwju/VXrEuwf7UM7kJ3GUJHANVAQXOJ7Xv9ei3Mcg5GAqLa+aort f8uvp+ehUKTJTCKIN3lB59fsaT4zTRfHTR/fSNxtjaNtWzmwOC+FAqkim8foTH8zGORv 9wNg== X-Gm-Message-State: AOAM533zRqJFix8Ci8J9RwpFKGk3Fu4GD+fMT2oOAPNBpc2/eYSegXsU fPmtmSGo7nrrEnhldwW70vHHvocwXyuJoQ== X-Google-Smtp-Source: ABdhPJxg/XhC8VWDNAAVjAXP26rzAXI6ryovuujZvCnQCFSCMDLBn/1xtrwWkubv+ib/xihHR8aG/Q== X-Received: by 2002:a37:48c7:: with SMTP id v190mr2857088qka.153.1595468941078; Wed, 22 Jul 2020 18:49:01 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:351c:83f3:acec:de38]) by smtp.gmail.com with ESMTPSA id t127sm1379762qkc.100.2020.07.22.18.49.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2020 18:49:00 -0700 (PDT) Date: Wed, 22 Jul 2020 21:48:59 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, chriscool@tuxfamily.org, gitster@pobox.com, szeder.dev@gmail.com Subject: [PATCH v2 1/4] list_objects_filter_options: introduce 'list_object_filter_config_name' Message-ID: <69605299bb38a678338365cf05178ce0735c7c67.1595468659.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In a subsequent commit, we will add configuration options that are specific to each kind of object filter, in which case it is handy to have a function that translates between 'enum list_objects_filter_choice' and an appropriate configuration-friendly string. Signed-off-by: Taylor Blau --- list-objects-filter-options.c | 23 +++++++++++++++++++++++ list-objects-filter-options.h | 6 ++++++ 2 files changed, 29 insertions(+) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 3553ad7b0a..92b408c0c8 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -15,6 +15,29 @@ static int parse_combine_filter( const char *arg, struct strbuf *errbuf); +const char *list_object_filter_config_name(enum list_objects_filter_choice c) +{ + switch (c) { + case LOFC_DISABLED: + /* we have no name for "no filter at all" */ + break; + case LOFC_BLOB_NONE: + return "blob:none"; + case LOFC_BLOB_LIMIT: + return "blob:limit"; + case LOFC_TREE_DEPTH: + return "tree"; + case LOFC_SPARSE_OID: + return "sparse:oid"; + case LOFC_COMBINE: + return "combine"; + case LOFC__COUNT: + /* not a real filter type; just the count of all filters */ + break; + } + BUG("list_object_filter_choice_name: invalid argument '%d'", c); +} + /* * Parse value of the argument to the "filter" keyword. * On the command line this looks like: diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index 73fffa4ad7..01767c3c96 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -17,6 +17,12 @@ enum list_objects_filter_choice { LOFC__COUNT /* must be last */ }; +/* + * Returns a configuration key suitable for describing the given object filter, + * e.g.: "blob:none", "combine", etc. + */ +const char *list_object_filter_config_name(enum list_objects_filter_choice c); + struct list_objects_filter_options { /* * 'filter_spec' is the raw argument value given on the command line From patchwork Thu Jul 23 01:49:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11679523 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 270556C1 for ; Thu, 23 Jul 2020 01:49:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 093DC20781 for ; Thu, 23 Jul 2020 01:49:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="Qwd6lEGb" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387528AbgGWBtH (ORCPT ); Wed, 22 Jul 2020 21:49:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729198AbgGWBtG (ORCPT ); Wed, 22 Jul 2020 21:49:06 -0400 Received: from mail-qv1-xf44.google.com (mail-qv1-xf44.google.com [IPv6:2607:f8b0:4864:20::f44]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B79BC0619DC for ; Wed, 22 Jul 2020 18:49:06 -0700 (PDT) Received: by mail-qv1-xf44.google.com with SMTP id el4so1925581qvb.13 for ; Wed, 22 Jul 2020 18:49:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=hENg6TEKnC0TvZ9BnPkb+T86J352jpTPJ5TV5UEEGho=; b=Qwd6lEGb53Z68opZESt5Uh5n9k1bi8Olv6hPJnhdliiKKQ5Bsf5Lv61hCcQXDgq79O UE/gfplNaCvoqkVXNkc/IECxo5VF0mLLzBIu+oiy8VCnIiaZf6qY3uLcbnhWBfHLQJcz gfmVs7h1xEJsd2DV0nC1qYPgmhP1biNU7tDYBhWLNULnFvEJqvK+LWZqAwovpvViI8aY IiZLyY9tr9nr6H03h2GUQTNPgUhQNYlLw5ItQm0GIdjdHIEyJOmCcKnrUhU/5NQxzn9f h8ZP5tt/n7GDk2xNw/SsjsvJEyvpA+LispjJLjUaIx/neZ6S1XV0GEmr99GvJC27dmez geBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=hENg6TEKnC0TvZ9BnPkb+T86J352jpTPJ5TV5UEEGho=; b=D+PVwOEH+YmJJy0KHkYiZdUOI2phGi6toFk0IVf6woKNW09m6F+0qFxGEgZl4UQ5Oi SKMk5he3SLYnTep3Y2/GCNP82qPHEmlDfCctVGd6PuvODzh1A6ZHUogO1TqkWfpj4g6Y 5w7FvvTtSDn2R18qmYW9oodXKFf/pJjAk7/W13TW52R4T7ruJ0n9IwNaffLb5Q0SPwB0 SiwwEPtH3n8VrWwjyG/jHXqb5xCvD491i5pXFdhf9SJT3KllAEkFGlfbogpJese08OiZ 3TA23OEE3I0KiyhDVPyhQjg90AVs283vjjXW+/quZo/pSaCo7hJgsYDb7o+AoM0Ph3sv Ecyw== X-Gm-Message-State: AOAM532LStlXHDiH7jBTY+4hMKu24pj8R00mBEURxlDnEwPS1Tzeqr+e QGkW+um2+PW2kWLOxGG3Gi3jFda9rUufxw== X-Google-Smtp-Source: ABdhPJyvm+4lalRTLUuTeEPFIyzVoSCJXG7vVRODkzT47jcBEkiLcCtA0g8/D/edxFePLXD0nI95MQ== X-Received: by 2002:a0c:a4e3:: with SMTP id x90mr2811187qvx.36.1595468945249; Wed, 22 Jul 2020 18:49:05 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:351c:83f3:acec:de38]) by smtp.gmail.com with ESMTPSA id z1sm1370628qkl.74.2020.07.22.18.49.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2020 18:49:04 -0700 (PDT) Date: Wed, 22 Jul 2020 21:49:03 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, chriscool@tuxfamily.org, gitster@pobox.com, szeder.dev@gmail.com Subject: [PATCH v2 2/4] upload-pack.c: allow banning certain object filter(s) Message-ID: <9fee52cb6dbc5eb7157a145eb6764b35b6a0e2f3.1595468659.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Git clients may ask the server for a partial set of objects, where the set of objects being requested is refined by one or more object filters. Server administrators can configure 'git upload-pack' to allow or ban these filters by setting the 'uploadpack.allowFilter' variable to 'true' or 'false', respectively. However, administrators using bitmaps may wish to allow certain kinds of object filters, but ban others. Specifically, they may wish to allow object filters that can be optimized by the use of bitmaps, while rejecting other object filters which aren't and represent a perceived performance degradation (as well as an increased load factor on the server). Allow configuring 'git upload-pack' to support object filters on a case-by-case basis by introducing two new configuration variables: - 'uploadpackfilter.allow' - 'uploadpackfilter..allow' where '' may be one of 'blobNone', 'blobLimit', 'tree', and so on. Setting the second configuration variable for any valid value of '' explicitly allows or disallows restricting that kind of object filter. If a client requests the object filter and the respective configuration value is not set, 'git upload-pack' will default to the value of 'uploadpackfilter.allow', which itself defaults to 'true' to maintain backwards compatibility. Note that this differs from 'uploadpack.allowfilter', which controls whether or not the 'filter' capability is advertised. Signed-off-by: Taylor Blau --- Documentation/config/uploadpack.txt | 12 +++++ t/t5616-partial-clone.sh | 24 +++++++++ upload-pack.c | 76 +++++++++++++++++++++++++++++ 3 files changed, 112 insertions(+) diff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt index ed1c835695..fffe8ac648 100644 --- a/Documentation/config/uploadpack.txt +++ b/Documentation/config/uploadpack.txt @@ -57,6 +57,18 @@ uploadpack.allowFilter:: If this option is set, `upload-pack` will support partial clone and partial fetch object filtering. +uploadpackfilter.allow:: + Provides a default value for unspecified object filters (see: the + below configuration variable). + Defaults to `true`. + +uploadpackfilter..allow:: + Explicitly allow or ban the object filter corresponding to + ``, where `` may be one of: `blob:none`, + `blob:limit`, `tree`, `sparse:oid`, or `combine`. If using + combined filters, both `combine` and all of the nested filter + kinds must be allowed. Defaults to `uploadpackfilter.allow`. + uploadpack.allowRefInWant:: If this option is set, `upload-pack` will support the `ref-in-want` feature of the protocol version 2 `fetch` command. This feature diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 8a27452a51..b196ee694c 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -235,6 +235,30 @@ test_expect_success 'implicitly construct combine: filter with repeated flags' ' test_cmp unique_types.expected unique_types.actual ' +test_expect_success 'upload-pack fails banned object filters' ' + test_config -C srv.bare uploadpackfilter.blob:none.allow false && + test_must_fail git clone --no-checkout --filter=blob:none \ + "file://$(pwd)/srv.bare" pc3 2>err && + test_i18ngrep "filter '\''blob:none'\'' not supported" err +' + +test_expect_success 'upload-pack fails banned combine object filters' ' + test_config -C srv.bare uploadpackfilter.allow false && + test_config -C srv.bare uploadpackfilter.combine.allow true && + test_config -C srv.bare uploadpackfilter.tree.allow true && + test_config -C srv.bare uploadpackfilter.blob:none.allow false && + test_must_fail git clone --no-checkout --filter=tree:1 \ + --filter=blob:none "file://$(pwd)/srv.bare" pc3 2>err && + test_i18ngrep "filter '\''blob:none'\'' not supported" err +' + +test_expect_success 'upload-pack fails banned object filters with fallback' ' + test_config -C srv.bare uploadpackfilter.allow false && + test_must_fail git clone --no-checkout --filter=blob:none \ + "file://$(pwd)/srv.bare" pc3 2>err && + test_i18ngrep "filter '\''blob:none'\'' not supported" err +' + test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' ' rm -rf src dst && git init src && diff --git a/upload-pack.c b/upload-pack.c index 951a2b23aa..61929977ab 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -88,6 +88,7 @@ struct upload_pack_data { enum allow_uor allow_uor; struct list_objects_filter_options filter_options; + struct string_list allowed_filters; struct packet_writer writer; @@ -103,6 +104,7 @@ struct upload_pack_data { unsigned no_progress : 1; unsigned use_include_tag : 1; unsigned allow_filter : 1; + unsigned allow_filter_fallback : 1; unsigned done : 1; /* v2 only */ unsigned allow_ref_in_want : 1; /* v2 only */ @@ -120,6 +122,7 @@ static void upload_pack_data_init(struct upload_pack_data *data) struct string_list deepen_not = STRING_LIST_INIT_DUP; struct string_list uri_protocols = STRING_LIST_INIT_DUP; struct object_array extra_edge_obj = OBJECT_ARRAY_INIT; + struct string_list allowed_filters = STRING_LIST_INIT_DUP; memset(data, 0, sizeof(*data)); data->symref = symref; @@ -131,6 +134,8 @@ static void upload_pack_data_init(struct upload_pack_data *data) data->deepen_not = deepen_not; data->uri_protocols = uri_protocols; data->extra_edge_obj = extra_edge_obj; + data->allowed_filters = allowed_filters; + data->allow_filter_fallback = 1; packet_writer_init(&data->writer, 1); data->keepalive = 5; @@ -147,6 +152,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data) string_list_clear(&data->deepen_not, 0); object_array_clear(&data->extra_edge_obj); list_objects_filter_release(&data->filter_options); + string_list_clear(&data->allowed_filters, 1); free((char *)data->pack_objects_hook); } @@ -983,6 +989,46 @@ static int process_deepen_not(const char *line, struct string_list *deepen_not, return 0; } +static int allows_filter_choice(struct upload_pack_data *data, + enum list_objects_filter_choice c) +{ + const char *key = list_object_filter_config_name(c); + struct string_list_item *item = string_list_lookup(&data->allowed_filters, + key); + if (item) + return (intptr_t) item->util; + return data->allow_filter_fallback; +} + +static struct list_objects_filter_options *banned_filter( + struct upload_pack_data *data, + struct list_objects_filter_options *opts) +{ + size_t i; + + if (!allows_filter_choice(data, opts->choice)) + return opts; + + if (opts->choice == LOFC_COMBINE) + for (i = 0; i < opts->sub_nr; i++) { + struct list_objects_filter_options *sub = &opts->sub[i]; + if (banned_filter(data, sub)) + return sub; + } + return NULL; +} + +static void die_if_using_banned_filter(struct upload_pack_data *data) +{ + struct list_objects_filter_options *banned = banned_filter(data, + &data->filter_options); + if (!banned) + return; + + die(_("git upload-pack: filter '%s' not supported"), + list_object_filter_config_name(banned->choice)); +} + static void receive_needs(struct upload_pack_data *data, struct packet_reader *reader) { @@ -1013,6 +1059,7 @@ static void receive_needs(struct upload_pack_data *data, die("git upload-pack: filtering capability not negotiated"); list_objects_filter_die_if_populated(&data->filter_options); parse_list_objects_filter(&data->filter_options, arg); + die_if_using_banned_filter(data); continue; } @@ -1170,6 +1217,32 @@ static int find_symref(const char *refname, const struct object_id *oid, return 0; } +static int parse_object_filter_config(const char *var, const char *value, + struct upload_pack_data *data) +{ + struct strbuf buf = STRBUF_INIT; + const char *sub, *key; + size_t sub_len; + + if (parse_config_key(var, "uploadpackfilter", &sub, &sub_len, &key)) + return 0; + + if (!sub) { + if (!strcmp(key, "allow")) + data->allow_filter_fallback = git_config_bool(var, value); + return 0; + } + + strbuf_add(&buf, sub, sub_len); + + if (!strcmp(key, "allow")) + string_list_insert(&data->allowed_filters, buf.buf)->util = + (void *)(intptr_t)git_config_bool(var, value); + + strbuf_release(&buf); + return 0; +} + static int upload_pack_config(const char *var, const char *value, void *cb_data) { struct upload_pack_data *data = cb_data; @@ -1209,6 +1282,8 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data) return git_config_string(&data->pack_objects_hook, var, value); } + parse_object_filter_config(var, value, data); + return parse_hide_refs_config(var, value, "uploadpack"); } @@ -1389,6 +1464,7 @@ static void process_args(struct packet_reader *request, if (data->allow_filter && skip_prefix(arg, "filter ", &p)) { list_objects_filter_die_if_populated(&data->filter_options); parse_list_objects_filter(&data->filter_options, p); + die_if_using_banned_filter(data); continue; } From patchwork Thu Jul 23 01:49:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11679525 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C3A9213A4 for ; Thu, 23 Jul 2020 01:49:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AA67520781 for ; Thu, 23 Jul 2020 01:49:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="2TB0DkNv" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387545AbgGWBtK (ORCPT ); Wed, 22 Jul 2020 21:49:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729198AbgGWBtK (ORCPT ); Wed, 22 Jul 2020 21:49:10 -0400 Received: from mail-qt1-x843.google.com (mail-qt1-x843.google.com [IPv6:2607:f8b0:4864:20::843]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5FEE5C0619DC for ; Wed, 22 Jul 2020 18:49:10 -0700 (PDT) Received: by mail-qt1-x843.google.com with SMTP id e7so3385959qti.1 for ; Wed, 22 Jul 2020 18:49:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ZICLeGvz8UZnLvIMCuBDtR6sPnwCgCu5fufpEMfmzyw=; b=2TB0DkNvd5yq5k67gtlQdp2zdOQ61IpUdTF82ePxME1waEFkkud37bVLCRfeNrkHVy w0drIkJ/VsGfjSB2K3q7woyClvpLjstqOXdkXurbeD8JugttqQu0UnZuUDeIrG+5ZPsQ aXfKrHK0yU5dwiOJyYYMrigtaqT7S3QgYAi0YLhztfHBjuIKIZbkw0qUrU2JZd4wCpMX xt04GkTYExrzQtQGHFprxiBelMXfO5kZkHkXtZUN8q6cUipBcPTYxFVRMkSmKZzWDYP5 wksMv+g9Td8i7pNEKE8UlT4vCs4eYhXjuVwqn3uECMrLUVRJ9dDBlfd3Dj/yxUkQGlF3 JJKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ZICLeGvz8UZnLvIMCuBDtR6sPnwCgCu5fufpEMfmzyw=; b=BFvhOaxApLwxNYDYpOTYmlRL2LiWUzUeM/iTtbkF2HhUD9TDHXvVq8fwfZEdtJXnOb VB+K0/hGZyUSwy2GKfPZcYtZJIQV8va2e/J46WlFaYpkuSPdMVLzcUUWQ4fVM33EzSm/ uuzfcdaZb2aRnH2CjsjXbZjgiWhzQebRD3U3lUgkncogLMSxgSNYIleitJFABrk85ZtF ze6GNhfBdRoNj61Z4Sx9cGpAjOfwO0Wz857/8nuFRVUR3HLRgQAIMdC/lzlmFkfVQtLr 2XoI91FewXfFkpe+G/azhL+SeOegti/3SOBsdVyYNpEaJxtkVhutm/AVmV2RfzSA8O/E 3goQ== X-Gm-Message-State: AOAM533Q1fIxXYktflzziMGqnnGztUf/ZrUNZR+Z/GmcIvSVTV99t4A+ jy7/ObH1T4ctDdEPfS6CYPymjmf13Wc9CQ== X-Google-Smtp-Source: ABdhPJxe4QKCJnSNTUjigH8NNSveklOI0km3+XGb1qWWoFfplI2aDeUW+toSv5QUAkm2prcRqvh+Hw== X-Received: by 2002:ac8:4e03:: with SMTP id c3mr2247443qtw.82.1595468949270; Wed, 22 Jul 2020 18:49:09 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:351c:83f3:acec:de38]) by smtp.gmail.com with ESMTPSA id x12sm1401923qta.67.2020.07.22.18.49.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2020 18:49:08 -0700 (PDT) Date: Wed, 22 Jul 2020 21:49:07 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, chriscool@tuxfamily.org, gitster@pobox.com, szeder.dev@gmail.com Subject: [PATCH v2 3/4] upload-pack.c: pass 'struct list_objects_filter_options *' Message-ID: <550e4e13f16016b219445376fdcd75da87d60c65.1595468659.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The 'allows_filter_choice' function used to take an 'enum list_objects_filter_choice', but in a future commit it will be more convenient for it to accept the whole 'struct list_objects_filter_options', for e.g., to inspect the value of '->tree_exclude_depth'. Signed-off-by: Taylor Blau --- upload-pack.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/upload-pack.c b/upload-pack.c index 61929977ab..48f10d21f6 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -990,9 +990,9 @@ static int process_deepen_not(const char *line, struct string_list *deepen_not, } static int allows_filter_choice(struct upload_pack_data *data, - enum list_objects_filter_choice c) + struct list_objects_filter_options *opts) { - const char *key = list_object_filter_config_name(c); + const char *key = list_object_filter_config_name(opts->choice); struct string_list_item *item = string_list_lookup(&data->allowed_filters, key); if (item) @@ -1006,7 +1006,7 @@ static struct list_objects_filter_options *banned_filter( { size_t i; - if (!allows_filter_choice(data, opts->choice)) + if (!allows_filter_choice(data, opts)) return opts; if (opts->choice == LOFC_COMBINE) From patchwork Thu Jul 23 01:49:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11679527 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B30E96C1 for ; Thu, 23 Jul 2020 01:49:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 95AF820771 for ; Thu, 23 Jul 2020 01:49:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="BNyNS7xc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387556AbgGWBtO (ORCPT ); Wed, 22 Jul 2020 21:49:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44580 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729198AbgGWBtO (ORCPT ); Wed, 22 Jul 2020 21:49:14 -0400 Received: from mail-qv1-xf41.google.com (mail-qv1-xf41.google.com [IPv6:2607:f8b0:4864:20::f41]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 40F51C0619DC for ; Wed, 22 Jul 2020 18:49:14 -0700 (PDT) Received: by mail-qv1-xf41.google.com with SMTP id el4so1925692qvb.13 for ; Wed, 22 Jul 2020 18:49:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=bNJ55miaRx/EShXoUMw2rdQnWBw6IOU1ZIF4ni4DCq0=; b=BNyNS7xc9WUGT4OZmQ8lXXaT5ypuijSvZVAwaDOVRfMQm5dbZxDYTDDk8j4XmbKQgF +DzV1sJ3qGJWQwAv1AZRdNfa10+2FMPCdqG4oWdra3w1AZDHMOiGH0VDZm/9EcGL2qIb HbtA/IiE35qtFiA90WhT9+j95gTRfdQeA1lrfYNOSZe5W0XxvLLakZ2rLZz9JsqgEb0y +FR3mHe9LQoTPXVXODNAZz3Bd4lNTuVpATTTl2bbqLmCz8gxeibjuLrDckvXrYu6VwtF djkzi8JQkdDXdbb4xPKYLJ+6otthb21nNNjQ8L8RTkTDywT1q5O62sfpvyPZSAY/6zPc V7SQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=bNJ55miaRx/EShXoUMw2rdQnWBw6IOU1ZIF4ni4DCq0=; b=O4gbp4gAWy+YJJjcyBS1I6LFp9v4YlNwO1HNJUoe07OTM/mat1tX2vMR91xCIOdL+w 0fPrWQ8ARBa/CoAo7AtHD7OnOaCPxZ/VKPLhrQOu0jq3vGhSEFVuF0OmQtkGA/bG9pQX Pg/fQPusHtUPn/xejmGDJ5BLszaYVA83vpnAwgBq9K0XCsdDRwXgeQWL0NXnpr+VYPmo fspkk1b+tCtiHIFleKrRsc+2y4JtSFKtqDtH5PYE4wAeGm85UL7Woz/bHJFFqAK0VzQu 1pps1nF5bHxSwoYcbOXluAP7Qmos1taUko6I/XaYgRass1XlZo9i1hNJwoi2MjQIxGud 7PGQ== X-Gm-Message-State: AOAM533k2+t3GcEha4wLqdmfE13K+Kp8zFODPDbDfFW4LhQoc7sG+SHQ 6T8piWt3eDPwyL2mXi+FYXDi7G6Yl0d7EA== X-Google-Smtp-Source: ABdhPJykxeB1Vc14LvDCBjtcNHp9KnQTH86rHANR/wdfiSrUUF9bMLn3Yi0KO1LpNN1MF/d9aKvKag== X-Received: by 2002:a0c:b315:: with SMTP id s21mr2763496qve.53.1595468952953; Wed, 22 Jul 2020 18:49:12 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:351c:83f3:acec:de38]) by smtp.gmail.com with ESMTPSA id b131sm1403239qkc.121.2020.07.22.18.49.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2020 18:49:12 -0700 (PDT) Date: Wed, 22 Jul 2020 21:49:11 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, chriscool@tuxfamily.org, gitster@pobox.com, szeder.dev@gmail.com Subject: [PATCH v2 4/4] upload-pack.c: introduce 'uploadpackfilter.tree.maxDepth' Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In b79cf959b2 (upload-pack.c: allow banning certain object filter(s), 2020-02-26), we introduced functionality to disallow certain object filters from being chosen from within 'git upload-pack'. Traditionally, administrators use this functionality to disallow filters that are known to perform slowly, for e.g., those that do not have bitmap-level filtering. In the past, the '--filter=tree:' was one such filter that does not have bitmap-level filtering support, and so was likely to be banned by administrators. However, in the previous couple of commits, we introduced bitmap-level filtering for the case when 'n' is equal to '0', i.e., as if we had a '--filter=tree:none' choice. While it would be sufficient to simply write $ git config uploadpackfilter.tree.allow true (since it would allow all values of 'n'), we would like to be able to allow this filter for certain values of 'n', i.e., those no greater than some pre-specified maximum. In order to do this, introduce a new configuration key, as follows: $ git config uploadpackfilter.tree.maxDepth where '' specifies the maximum allowed value of 'n' in the filter 'tree:n'. Administrators who wish to allow for only the value '0' can write: $ git config uploadpackfilter.tree.allow true $ git config uploadpackfilter.tree.maxDepth 0 which allows '--filter=tree:0', but no other values. Unfortunately, since the tree depth is an unsigned long, we can't use, say, -1 as a sentinel value, and so we must also keep track of "have we set this" as well as "to what value". Signed-off-by: Taylor Blau --- Documentation/config/uploadpack.txt | 6 ++++++ t/t5616-partial-clone.sh | 8 +++++++ upload-pack.c | 33 ++++++++++++++++++++++++++--- 3 files changed, 44 insertions(+), 3 deletions(-) diff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt index fffe8ac648..ee7b3ac94f 100644 --- a/Documentation/config/uploadpack.txt +++ b/Documentation/config/uploadpack.txt @@ -69,6 +69,12 @@ uploadpackfilter..allow:: combined filters, both `combine` and all of the nested filter kinds must be allowed. Defaults to `uploadpackfilter.allow`. +uploadpackfilter.tree.maxDepth:: + Only allow `--filter=tree=` when `n` is no more than the value of + `uploadpackfilter.tree.maxDepth`. If set, this also implies + `uploadpackfilter.tree.allow=true`, unless this configuration + variable had already been set. Has no effect if unset. + uploadpack.allowRefInWant:: If this option is set, `upload-pack` will support the `ref-in-want` feature of the protocol version 2 `fetch` command. This feature diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index b196ee694c..4292a644d7 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -259,6 +259,14 @@ test_expect_success 'upload-pack fails banned object filters with fallback' ' test_i18ngrep "filter '\''blob:none'\'' not supported" err ' +test_expect_success 'upload-pack limits tree depth filters' ' + test_config -C srv.bare uploadpackfilter.allow false && + test_config -C srv.bare uploadpackfilter.tree.allow true && + test_config -C srv.bare uploadpackfilter.tree.maxDepth 0 && + test_must_fail ok=sigpipe git clone --no-checkout --filter=tree:1 \ + "file://$(pwd)/srv.bare" pc3 +' + test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' ' rm -rf src dst && git init src && diff --git a/upload-pack.c b/upload-pack.c index 48f10d21f6..47cdaae265 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -105,6 +105,7 @@ struct upload_pack_data { unsigned use_include_tag : 1; unsigned allow_filter : 1; unsigned allow_filter_fallback : 1; + unsigned long tree_filter_max_depth; unsigned done : 1; /* v2 only */ unsigned allow_ref_in_want : 1; /* v2 only */ @@ -136,6 +137,7 @@ static void upload_pack_data_init(struct upload_pack_data *data) data->extra_edge_obj = extra_edge_obj; data->allowed_filters = allowed_filters; data->allow_filter_fallback = 1; + data->tree_filter_max_depth = ULONG_MAX; packet_writer_init(&data->writer, 1); data->keepalive = 5; @@ -995,8 +997,17 @@ static int allows_filter_choice(struct upload_pack_data *data, const char *key = list_object_filter_config_name(opts->choice); struct string_list_item *item = string_list_lookup(&data->allowed_filters, key); + int allowed = -1; if (item) - return (intptr_t) item->util; + allowed = (intptr_t) item->util; + + if (allowed != 0 && + opts->choice == LOFC_TREE_DEPTH && + opts->tree_exclude_depth > data->tree_filter_max_depth) + return 0; + + if (allowed > -1) + return allowed; return data->allow_filter_fallback; } @@ -1022,11 +1033,18 @@ static void die_if_using_banned_filter(struct upload_pack_data *data) { struct list_objects_filter_options *banned = banned_filter(data, &data->filter_options); + struct strbuf buf = STRBUF_INIT; if (!banned) return; - die(_("git upload-pack: filter '%s' not supported"), - list_object_filter_config_name(banned->choice)); + strbuf_addf(&buf, _("filter '%s' not supported"), + list_object_filter_config_name(banned->choice)); + if (banned->choice == LOFC_TREE_DEPTH && + data->tree_filter_max_depth != ULONG_MAX) + strbuf_addf(&buf, _(" (maximum depth: %lu, but got: %lu)"), + data->tree_filter_max_depth, + banned->tree_exclude_depth); + die("%s", buf.buf); } static void receive_needs(struct upload_pack_data *data, @@ -1238,6 +1256,15 @@ static int parse_object_filter_config(const char *var, const char *value, if (!strcmp(key, "allow")) string_list_insert(&data->allowed_filters, buf.buf)->util = (void *)(intptr_t)git_config_bool(var, value); + else if (!strcmp(buf.buf, "tree") && !strcmp(key, "maxdepth")) { + if (!value) { + strbuf_release(&buf); + return config_error_nonbool(var); + } + string_list_insert(&data->allowed_filters, buf.buf)->util = + (void *)(intptr_t)1; + data->tree_filter_max_depth = git_config_ulong(var, value); + } strbuf_release(&buf); return 0;