From patchwork Mon Mar 28 19:11:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 12794138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6E04C433F5 for ; Mon, 28 Mar 2022 19:11:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245484AbiC1TNd (ORCPT ); Mon, 28 Mar 2022 15:13:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57790 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229878AbiC1TNd (ORCPT ); Mon, 28 Mar 2022 15:13:33 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A419766AFD for ; Mon, 28 Mar 2022 12:11:47 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id x186-20020a627cc3000000b004fa939658c5so8921455pfc.4 for ; Mon, 28 Mar 2022 12:11:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=RRRNOeWCzRnmh+kYHqpNKr2pf5a0v9m/RNnrhzP1c0o=; b=nnk2rE8dc9OgTB0VQ2OMtwvjfSorAmKB/jXU7lrFnLVI77XsdPoaAIbrJdu/adFuGP NJ3bGYDWRpeQHVXOTMD5PcvLsU66pmgwp1wuSJ2HZUFrDKSDpZcuiXlqi/2GysURylca 28l6tmQ02TiMhX4CsaI6MAm6aZljKGVSNStIsnZXUxIoqw5r5IkiV6hudDZptry7TNHh jKxqFy+QyuE4IBM8qZ06w4Du/twGeD2ApzLHiZBr5AzM7hi/zt33qzu943RL8AWefgO8 OCDEiybKHN/gisX7F1h5f8HzDDCqPgw3n1AwIgg4zvLyFjXu6r46UDZ2ALh7Oe9XvP0p 4sqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=RRRNOeWCzRnmh+kYHqpNKr2pf5a0v9m/RNnrhzP1c0o=; b=j6dukliUJ8q38QD/fmhGNbAlZ4zCyAIm9T7kIJl9rXx4cqjiVTSLgHxh6tKqyud9HT 4xn19h54SUh7gXZYzZ/MbTZGVMY7rDkgF+cHckfz4RXhPUkFnXRdoJ0I4uy5CcRHJB2X V6VbzwV2g4FCc5ApgIRm0uDCRlgaGIARD8dU6odbJf0TMfGMBhmRNUp3XZDWQCG2t42i K36kCnJ7bIaLQ4ztU5nKqs3P4JV8dzGuXwBQ3RK4yiZ9WHHqFA6P6i63rDj7AAxERCvK +sNRlqUFlYarF/OtxZXwBXSGvJjjNYXGX+jYn4NRtu+ZTJIysMM7oAST12SpfE5bq5lj Sm4A== X-Gm-Message-State: AOAM5322/9EoMRgqNNCO0dTlLyE5syGN7rmF6mHHB3x3Ufg7fye/Rwkh K67IUVqw6Q8TES8ggdCFsLLcGdnFwt4ShzCEOnqe83g1He/A2KUtJbXoYS4tIKCLU7YRr16Cqrg sbkDa+1JD1KMHApli4H6+98UqyXusoWZ3ZF470SYyNFWtn/yxoOi7d7/paR7L6z8OFA== X-Google-Smtp-Source: ABdhPJygirnw9uvjoq1EOImfyZp3CDzcTqV2XA5iCC57gJU9Vgq2bhjCWbW3m457rTIqy43xv3X9EAtoVB4g+Qw= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:90b:a56:b0:1c7:327d:9ac with SMTP id gw22-20020a17090b0a5600b001c7327d09acmr661237pjb.108.1648494707064; Mon, 28 Mar 2022 12:11:47 -0700 (PDT) Date: Mon, 28 Mar 2022 19:11:10 +0000 In-Reply-To: <20220328191112.3092139-1-calvinwan@google.com> Message-Id: <20220328191112.3092139-2-calvinwan@google.com> Mime-Version: 1.0 References: <20220208235631.732466-1-calvinwan@google.com> <20220328191112.3092139-1-calvinwan@google.com> X-Mailer: git-send-email 2.33.0.664.g0785eb7698 Subject: [PATCH v3 1/3] fetch-pack: refactor packet writing and fetch options From: Calvin Wan To: git@vger.kernel.org Cc: jonathantanmy@google.com, philipoakley@iee.email, johncai86@gmail.com, Calvin Wan Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A subsequent commit will need to write capabilities for another command, so refactor write_fetch_command_and_capabilities() to be able to used by both fetch and future command. Move fetch options set in `FETCH_CHECK_LOCAL` from the fetch state machine to above the state machine so it is set by default. The initial state of the state machine is always `FETCH_CHECK_LOCAL` so this does not affect any current functionality. This change prepares for a subsequent commit that doesn't need to check the local state, but still requires those options to be set before sending the fetch request. --- fetch-pack.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index 87657907e7..b709a61baf 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1237,13 +1237,13 @@ static int add_haves(struct fetch_negotiator *negotiator, return haves_added; } -static void write_fetch_command_and_capabilities(struct strbuf *req_buf, - const struct string_list *server_options) +static void write_command_and_capabilities(struct strbuf *req_buf, + const struct string_list *server_options, const char* command) { const char *hash_name; - if (server_supports_v2("fetch", 1)) - packet_buf_write(req_buf, "command=fetch"); + if (server_supports_v2(command, 1)) + packet_buf_write(req_buf, "command=%s", command); if (server_supports_v2("agent", 0)) packet_buf_write(req_buf, "agent=%s", git_user_agent_sanitized()); if (advertise_sid && server_supports_v2("session-id", 0)) @@ -1279,7 +1279,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, int done_sent = 0; struct strbuf req_buf = STRBUF_INIT; - write_fetch_command_and_capabilities(&req_buf, args->server_options); + write_command_and_capabilities(&req_buf, args->server_options, "fetch"); if (args->use_thin_pack) packet_buf_write(&req_buf, "thin-pack"); @@ -1598,18 +1598,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, reader.me = "fetch-pack"; } + /* v2 supports these by default */ + allow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1; + use_sideband = 2; + if (args->depth > 0 || args->deepen_since || args->deepen_not) + args->deepen = 1; + while (state != FETCH_DONE) { switch (state) { case FETCH_CHECK_LOCAL: sort_ref_list(&ref, ref_compare_name); QSORT(sought, nr_sought, cmp_ref_by_name); - /* v2 supports these by default */ - allow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1; - use_sideband = 2; - if (args->depth > 0 || args->deepen_since || args->deepen_not) - args->deepen = 1; - /* Filter 'ref' by 'sought' and those that aren't local */ mark_complete_and_common_ref(negotiator, args, &ref); filter_refs(args, &ref, sought, nr_sought); @@ -2060,7 +2060,7 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips, int received_ready = 0; strbuf_reset(&req_buf); - write_fetch_command_and_capabilities(&req_buf, server_options); + write_command_and_capabilities(&req_buf, server_options, "fetch"); packet_buf_write(&req_buf, "wait-for-done"); From patchwork Mon Mar 28 19:11:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 12794137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BC5BC433EF for ; Mon, 28 Mar 2022 19:11:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245489AbiC1TNg (ORCPT ); Mon, 28 Mar 2022 15:13:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229878AbiC1TNf (ORCPT ); Mon, 28 Mar 2022 15:13:35 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0ED766AFD for ; Mon, 28 Mar 2022 12:11:53 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id r11-20020a63440b000000b0038068f34b0cso7379293pga.0 for ; Mon, 28 Mar 2022 12:11:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=zODZQjxc0KTlnGF7R5lE+G3y2toI8Hx/er5T5WWzE78=; b=XoCoLFFIP5s1TDerIFZm5BFM+nv23X6nkgZwLbTekFb7WZODxVLvHnGk4HL8XUpVhY DtXhKU53YmvahZjkS8ncoHV4VBdhoL3tZs16rdYVROIXLbUELYnoPWHk4PcaUv+7E1VK d5jPXG52FmIZS9GjD0VCQ5SROR4OYciyf+m3IUDsCun7Mx0a8pTpN6l8uUt2VAG1LlGz LM/lWMuz1fvwJpb/R0fxI1GQLYvkMMbnEtcpSW7sJ6v8naFVo+Wf62r45CO1+qGqAMON vWpR7oHgooFbAcXeZocPRnmIKwwLOCQKtY6snhCP32mvNTDjYGpNK/p5EwAh7ONOwXNN q+zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=zODZQjxc0KTlnGF7R5lE+G3y2toI8Hx/er5T5WWzE78=; b=rWUfhpnMxVhaGaIHxvJFg1lnj+tWA8byniO8r1bB8QWjVCX9ny9t1PBB0lU5i/qzfM 7AzU+iCY0Gdq6iFNdUI0WtIWn2RbIUnLk5SBBqcukCbbJM+8MYY6SuxFVCDGFWufvEZ5 OLAlAB5auoEQm4dwDu/S6j5+pLcDCa43qYM39m0BRHAFgfeOpfFVOh7vsNuZ/bkk84yQ RubUW2lAeAfQWyQN75OK5zIJCXDZZJuX/kkSoZf9L5/JCWxqlYb5332sYRk+jIQPcZDu 6gKHQF50sIQ0NmRMVTjZcuj+qGSfZ+tGWPJ6C+9lv4vasL6ahI+5N0bhxB3+aW8spGk/ omEA== X-Gm-Message-State: AOAM532ZBqeKxDKhpKFzkWm/2dsyNP00qbGPZ7xRs0unn5j2v4Envpyx 7n154fUNA21jA5DQNh99p8poXJcoSasmLUs3EzcPxuDxztuaoZ/fqaxLoqQH1kUkcNzRHPNhwiw Vm61Dx8jV9ByZZHMBiENuQspbAItpljGh1EUQTxY9yBcV50n666JSXXWwL+pJPHnB1w== X-Google-Smtp-Source: ABdhPJxu7thJJsQ29CsG3NAHjCOJOhQQCPTNtajZu5YRrC3SQhkrO6pumi8DF2aCZGiffTheRmN6iDqgD5zlBII= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:90a:5407:b0:1bf:43ce:f11b with SMTP id z7-20020a17090a540700b001bf43cef11bmr625941pjh.31.1648494713269; Mon, 28 Mar 2022 12:11:53 -0700 (PDT) Date: Mon, 28 Mar 2022 19:11:11 +0000 In-Reply-To: <20220328191112.3092139-1-calvinwan@google.com> Message-Id: <20220328191112.3092139-3-calvinwan@google.com> Mime-Version: 1.0 References: <20220208235631.732466-1-calvinwan@google.com> <20220328191112.3092139-1-calvinwan@google.com> X-Mailer: git-send-email 2.33.0.664.g0785eb7698 Subject: [PATCH v3 2/3] transfer.advertiseObjectInfo: add object-info config From: Calvin Wan To: git@vger.kernel.org Cc: jonathantanmy@google.com, philipoakley@iee.email, johncai86@gmail.com, Calvin Wan Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Currently, object-info is always advertised by the server. Add a config option to optionally advertise it. A subsequent patch uses this option for testing. --- Documentation/config/transfer.txt | 4 ++++ protocol-caps.c | 11 +++++++++++ protocol-caps.h | 6 ++++++ serve.c | 2 +- 4 files changed, 22 insertions(+), 1 deletion(-) diff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt index b49429eb4d..14892a3155 100644 --- a/Documentation/config/transfer.txt +++ b/Documentation/config/transfer.txt @@ -77,3 +77,7 @@ transfer.unpackLimit:: transfer.advertiseSID:: Boolean. When true, client and server processes will advertise their unique session IDs to their remote counterpart. Defaults to false. + +transfer.advertiseObjectInfo:: + Boolean. When true or not set, server processes will advertise its + ability to accept `object-info` command requests \ No newline at end of file diff --git a/protocol-caps.c b/protocol-caps.c index bbde91810a..f290e3cca2 100644 --- a/protocol-caps.c +++ b/protocol-caps.c @@ -8,6 +8,9 @@ #include "object-store.h" #include "string-list.h" #include "strbuf.h" +#include "config.h" + +static int advertise_object_info = -1; struct requested_info { unsigned size : 1; @@ -111,3 +114,11 @@ int cap_object_info(struct repository *r, struct packet_reader *request) return 0; } + +int object_info_advertise(struct repository *r, struct strbuf *value) +{ + if (advertise_object_info == -1 && + git_config_get_bool("transfer.advertiseObjectInfo", &advertise_object_info)) + advertise_object_info = 0; + return advertise_object_info; +} diff --git a/protocol-caps.h b/protocol-caps.h index 15c4550360..36f7a46b37 100644 --- a/protocol-caps.h +++ b/protocol-caps.h @@ -5,4 +5,10 @@ struct repository; struct packet_reader; int cap_object_info(struct repository *r, struct packet_reader *request); +/* + * Advertises object-info capability if "objectinfo.advertise" is either + * set to true or not set + */ +int object_info_advertise(struct repository *r, struct strbuf *value); + #endif /* PROTOCOL_CAPS_H */ diff --git a/serve.c b/serve.c index b3fe9b5126..fb4ad367a7 100644 --- a/serve.c +++ b/serve.c @@ -133,7 +133,7 @@ static struct protocol_capability capabilities[] = { }, { .name = "object-info", - .advertise = always_advertise, + .advertise = object_info_advertise, .command = cap_object_info, }, }; From patchwork Mon Mar 28 19:11:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 12794139 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEA9CC433F5 for ; Mon, 28 Mar 2022 19:12:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229878AbiC1TNl (ORCPT ); Mon, 28 Mar 2022 15:13:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245492AbiC1TNk (ORCPT ); Mon, 28 Mar 2022 15:13:40 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F62066C98 for ; Mon, 28 Mar 2022 12:11:57 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id om8-20020a17090b3a8800b001c68e7ccd5fso114721pjb.9 for ; Mon, 28 Mar 2022 12:11:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=SoZWlW6d4S9YdRGFmXQRr0LsFS29a72V/Pp9Fe5Ntz4=; b=NXiw5N/KkrZX2umMFMWJRst61Y46EMzQcd2dXDSbRMoyXL1KYi95EjAbX5Sa83etSV 7beNhP2L1q/f9mRh95V3GbYHWoqBZoMqEovK+4NyKNbHwN6XRVg3W06X7WexC8l+qzar 7VP+UKJDezqZA/SOLnIVHUpXExztIfCwZMAk2WFfYNuli6CYXZSMU1XE1j+GyVd8JGXu OXetOMs2JAOeL1nfTRmvsz8ON4a6hwLV340Jxeh103ur9u67RApcjcRjpszP6rYJaMI4 dkRO5XET9TnM/rt+KfFmCfsWHOY/MpEtIXPpnUkqtyvDkrs3EJ/XOYYi11HH1iJlmPEl 5swQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=SoZWlW6d4S9YdRGFmXQRr0LsFS29a72V/Pp9Fe5Ntz4=; b=cLtr3B+T3pJckrDnNyb6wl2uWCV7h4+fFDWHvlJnT6DsYP5djgRyMhVVaGl21+y7Pp oIT76qk2WWPQv6C3feG/I5ANk5qAR0Judlp40Q11Q1siPuo7M8YLtSI5m5K4c/Bh5pVK 6lkbZi4wQtszsuQZ4+nmk6yEANhyNoOpLb2fTf9KhwOu4w3Dl0mgNPlbfuziPCxEM7Ng 2WkYZiaOa/CyrB5LErRiWoVCSzHBXj6ij/3T+FY4L6f01LuEQrtI1w4xJFcbFGHI4BuM IdRTGPIPX8oABSBacb5pQM/VMfoZPTTK7nCe2ox+jcFb3JgkKarCfv5lIoZsG7Corxc4 MUcA== X-Gm-Message-State: AOAM530S9hkcCRK9n2qFNINT1Eiq1CxKxZIrJ37m8bpf59ORm84D7fvI VQyxcqGqh9AOvhyXbjZB/0sZRhR7qYbnDosgGOrTirjXeuofARovGael0Kj68HirYCY641beA6s QKjHCFlpht8S+s18puvNgfRdA9uLdZxlS328s/dXgMCjcb4teesmUhqFuAEPHdrHKdg== X-Google-Smtp-Source: ABdhPJyph0ZJbFt5HQO2ahmFx1ZVt+60UD38kney9bJQqROvYkKF+fBZRogVw9uZVjKtlmPAQ6m2ffr2oVuaryU= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:90b:1b10:b0:1c9:cf93:c57 with SMTP id nu16-20020a17090b1b1000b001c9cf930c57mr579174pjb.154.1648494716721; Mon, 28 Mar 2022 12:11:56 -0700 (PDT) Date: Mon, 28 Mar 2022 19:11:12 +0000 In-Reply-To: <20220328191112.3092139-1-calvinwan@google.com> Message-Id: <20220328191112.3092139-4-calvinwan@google.com> Mime-Version: 1.0 References: <20220208235631.732466-1-calvinwan@google.com> <20220328191112.3092139-1-calvinwan@google.com> X-Mailer: git-send-email 2.33.0.664.g0785eb7698 Subject: [PATCH v3 3/3] object-info: add option for retrieving object info From: Calvin Wan To: git@vger.kernel.org Cc: jonathantanmy@google.com, philipoakley@iee.email, johncai86@gmail.com, Calvin Wan Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Sometimes it is useful to get information about an object without having to download it completely. The server logic has already been implemented as “a2ba162cda (object-info: support for retrieving object info, 2021-04-20)”. This patch implements the client option for it. Currently, only 'size' is supported, however future patches can implement additional options. Add ‘--object-info’ option to fetch. This option allows the client to make an object-info command request to a server that supports protocol v2. If the server is v2, but does not allow for the object-info command request, fetch the objects as if it were a standard fetch (however without changing any refs). Therefore, hook `fetch object-info` into transport_fetch_refs() to easily fallback if needed. A previous patch added the `transfer.advertiseObjectInfo` config option, of which this patch utilizes to test the fallback. --- Documentation/fetch-options.txt | 5 ++ Documentation/git-fetch.txt | 1 + builtin/fetch.c | 36 ++++++++- fetch-pack.c | 42 +++++++++- fetch-pack.h | 9 +++ t/t5583-fetch-object-info.sh | 138 ++++++++++++++++++++++++++++++++ transport-helper.c | 8 +- transport-internal.h | 1 + transport.c | 75 ++++++++++++++++- transport.h | 9 +++ 10 files changed, 315 insertions(+), 9 deletions(-) create mode 100755 t/t5583-fetch-object-info.sh diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt index f903683189..f1ca95c32d 100644 --- a/Documentation/fetch-options.txt +++ b/Documentation/fetch-options.txt @@ -299,3 +299,8 @@ endif::git-pull[] -6:: --ipv6:: Use IPv6 addresses only, ignoring IPv4 addresses. + +--object-info=[]:: + Fetches only the relevant information passed in arguments from a + specific remote and set of object ids. Object 'size' is currently + the only supported argument. diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt index 550c16ca61..a13d2ba7ad 100644 --- a/Documentation/git-fetch.txt +++ b/Documentation/git-fetch.txt @@ -13,6 +13,7 @@ SYNOPSIS 'git fetch' [] 'git fetch' --multiple [] [( | )...] 'git fetch' --all [] +'git fetch' --object-info=[] [] DESCRIPTION diff --git a/builtin/fetch.c b/builtin/fetch.c index 4d12c2fd4d..0b21bebfca 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -29,6 +29,9 @@ #include "commit-graph.h" #include "shallow.h" #include "worktree.h" +#include "protocol.h" +#include "pkt-line.h" +#include "connect.h" #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000) @@ -37,6 +40,7 @@ static const char * const builtin_fetch_usage[] = { N_("git fetch [] "), N_("git fetch --multiple [] [( | )...]"), N_("git fetch --all []"), + N_("git fetch --object-info=[] []"), NULL }; @@ -86,6 +90,7 @@ static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP; static int fetch_write_commit_graph = -1; static int stdin_refspecs = 0; static int negotiate_only; +static struct string_list object_info_options = STRING_LIST_INIT_NODUP; static int git_fetch_config(const char *k, const char *v, void *cb) { @@ -221,6 +226,8 @@ static struct option builtin_fetch_options[] = { N_("write the commit-graph after fetching")), OPT_BOOL(0, "stdin", &stdin_refspecs, N_("accept refspecs from stdin")), + OPT_STRING_LIST(0, "object-info", &object_info_options, N_("option"), + N_("command request arguments")), OPT_END() }; @@ -2087,6 +2094,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) struct remote *remote = NULL; int result = 0; int prune_tags_ok = 1; + struct oid_array object_info_oids = OID_ARRAY_INIT; packet_trace_identity("fetch"); @@ -2168,7 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) if (dry_run) write_fetch_head = 0; - if (all) { + if (object_info_options.nr > 0) { + if (argc == 0 || argc == 1) { + die(_("must supply remote and object ids when using --object-info")); + } else { + struct object_id oid; + remote = remote_get(argv[0]); + for (i = 1; i < argc; i++) { + if (get_oid(argv[i], &oid)) + return error(_("malformed object id '%s'"), argv[i]); + oid_array_append(&object_info_oids, &oid); + } + } + } else if (all) { if (argc == 1) die(_("fetch --all does not take a repository argument")); else if (argc > 1) @@ -2199,7 +2219,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) } } - if (negotiate_only) { + if (object_info_options.nr > 0) { + if (!remote) + die(_("must supply remote when using --object-info")); + gtransport = prepare_transport(remote, 1); + if (gtransport->smart_options) { + gtransport->smart_options->object_info = 1; + gtransport->smart_options->object_info_oids = &object_info_oids; + gtransport->smart_options->object_info_options = &object_info_options; + } + if (server_options.nr) + gtransport->server_options = &server_options; + result = transport_fetch_refs(gtransport, NULL); + } else if (negotiate_only) { struct oidset acked_commits = OIDSET_INIT; struct oidset_iter iter; const struct object_id *oid; diff --git a/fetch-pack.c b/fetch-pack.c index b709a61baf..36e3d1c5d0 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1269,6 +1269,27 @@ static void write_command_and_capabilities(struct strbuf *req_buf, packet_buf_delim(req_buf); } +void send_object_info_request(int fd_out, struct object_info_args *args) +{ + struct strbuf req_buf = STRBUF_INIT; + int i; + + write_command_and_capabilities(&req_buf, args->server_options, "object-info"); + + if (string_list_has_string(args->object_info_options, "size")) + packet_buf_write(&req_buf, "size"); + + for (i = 0; i < args->oids->nr; i++) { + packet_buf_write(&req_buf, "oid %s\n", oid_to_hex(&args->oids->oid[i])); + } + + packet_buf_flush(&req_buf); + if (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0) + die_errno(_("unable to write request to remote")); + + strbuf_release(&req_buf); +} + static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, struct fetch_pack_args *args, const struct ref *wants, struct oidset *common, @@ -1604,6 +1625,10 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, if (args->depth > 0 || args->deepen_since || args->deepen_not) args->deepen = 1; + if (args->object_info) { + state = FETCH_SEND_REQUEST; + } + while (state != FETCH_DONE) { switch (state) { case FETCH_CHECK_LOCAL: @@ -1613,7 +1638,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, /* Filter 'ref' by 'sought' and those that aren't local */ mark_complete_and_common_ref(negotiator, args, &ref); filter_refs(args, &ref, sought, nr_sought); - if (everything_local(args, &ref)) + if (!args->object_info && everything_local(args, &ref)) state = FETCH_DONE; else state = FETCH_SEND_REQUEST; @@ -1999,8 +2024,19 @@ struct ref *fetch_pack(struct fetch_pack_args *args, } args->connectivity_checked = 1; } - - update_shallow(args, sought, nr_sought, &si); + + if (args->object_info) { + struct ref *ref_cpy_reader = ref_cpy; + unsigned long size = 0; + while (ref_cpy_reader) { + oid_object_info(the_repository, &(ref_cpy_reader->old_oid), &size); + printf("%s %li\n", oid_to_hex(&(ref_cpy_reader->old_oid)), size); + ref_cpy_reader = ref_cpy_reader->next; + } + } + else { + update_shallow(args, sought, nr_sought, &si); + } cleanup: clear_shallow_info(&si); oid_array_clear(&shallows_scratch); diff --git a/fetch-pack.h b/fetch-pack.h index 7f94a2a583..23e69bb218 100644 --- a/fetch-pack.h +++ b/fetch-pack.h @@ -42,6 +42,7 @@ struct fetch_pack_args { unsigned update_shallow:1; unsigned reject_shallow_remote:1; unsigned deepen:1; + unsigned object_info:1; /* * Indicate that the remote of this request is a promisor remote. The @@ -68,6 +69,12 @@ struct fetch_pack_args { unsigned connectivity_checked:1; }; +struct object_info_args { + const struct string_list *object_info_options; + const struct string_list *server_options; + const struct oid_array *oids; +}; + /* * sought represents remote references that should be updated from. * On return, the names that were found on the remote will have been @@ -101,4 +108,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips, */ int report_unmatched_refs(struct ref **sought, int nr_sought); +void send_object_info_request(int fd_out, struct object_info_args *args); + #endif diff --git a/t/t5583-fetch-object-info.sh b/t/t5583-fetch-object-info.sh new file mode 100755 index 0000000000..e5db6d953c --- /dev/null +++ b/t/t5583-fetch-object-info.sh @@ -0,0 +1,138 @@ +#!/bin/sh + +test_description='test git fetch object-info version 2' + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test fetch object-info with 'git://' transport + +. "$TEST_DIRECTORY"/lib-git-daemon.sh +start_git_daemon --export-all --enable=receive-pack +daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent + + +test_expect_success 'create repo to be served by git-daemon' ' + git init "$daemon_parent" && + test_commit -C "$daemon_parent" message1 a.txt +' + +test_expect_success 'fetch object-info with git:// using protocol v2' ' + ( + cd "$daemon_parent" && + object_id=$(git rev-parse message1:a.txt) && + length=$(wc -c expect && + git -c protocol.version=2 fetch --object-info=size "$GIT_DAEMON_URL/parent" "$object_id" >actual && + test_cmp expect actual + ) +' +stop_git_daemon + +# Test protocol v2 with 'http://' transport + +. "$TEST_DIRECTORY"/lib-httpd.sh +start_httpd + +test_expect_success 'create repo to be served by http:// transport' ' + git init "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" config http.receivepack true && + test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" message1 a.txt && + test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" message2 b.txt +' + +test_expect_success 'fetch object-info with http:// using protocol v2' ' + ( + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + object_id=$(git rev-parse message1:a.txt) && + length=$(wc -c expect && + git -c protocol.version=2 fetch --object-info=size "$HTTPD_URL/smart/http_parent" "$object_id" >actual && + test_cmp expect actual + ) +' + +test_expect_success 'fetch object-info for multiple objects' ' + ( + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + object_id1=$(git rev-parse message1:a.txt) && + object_id2=$(git rev-parse message2:b.txt) && + length1=$(wc -c expect && + printf "%s %d\n" "$object_id2" "$length2" >>expect && + git -c protocol.version=2 fetch --object-info=size "$HTTPD_URL/smart/http_parent" "$object_id1" "$object_id2" >actual && + test_cmp expect actual + ) +' + +test_expect_success 'fetch object-info fallbacks to standard fetch if object-info is not supported by the server' ' + ( + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + object_id=$(git rev-parse message1:a.txt) && + length=$(wc -c expect && + git config objectinfo.advertise false && + git -c protocol.version=2 fetch --object-info=size "$HTTPD_URL/smart/http_parent" "$object_id" >actual && + test_cmp expect actual + ) +' + +test_expect_success 'fetch object-info fails on server with legacy protocol' ' + ( + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + object_id=$(git rev-parse message1:a.txt) && + test_must_fail git -c protocol.version=0 fetch --object-info=size "$HTTPD_URL/smart/http_parent" "$object_id" 2>err && + test_i18ngrep "object-info requires protocol v2" err + ) +' + +test_expect_success 'fetch object-info fails on malformed OID' ' + ( + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + malformed_object_id="this_id_is_not_valid" && + test_must_fail git -c protocol.version=2 fetch --object-info=size "$HTTPD_URL/smart/http_parent" "$malformed_object_id" 2>err && + test_i18ngrep "malformed object id '$malformed_object_id'" err + ) +' + +test_expect_success 'fetch object-info fails on missing OID' ' + git clone "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" missing_oid_repo && + test_commit -C missing_oid_repo message3 c.txt && + ( + cd missing_oid_repo && + object_id=$(git rev-parse message3:c.txt) && + test_must_fail env GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch --object-info=size "$HTTPD_URL/smart/http_parent" "$object_id" 2>err && + test_i18ngrep "fatal: remote error: upload-pack: not our ref $object_id" err + ) +' + +# Test fetch object-info with 'file://' transport + +test_expect_success 'create repo to be served by file:// transport' ' + git init server && + test_commit -C server message1 a.txt && + git -C server config protocol.version 2 +' + +test_expect_success 'fetch object-info with file:// using protocol v2' ' + ( + cd server && + object_id=$(git rev-parse message1:a.txt) && + length=$(wc -c expect && + git -c protocol.version=2 fetch --object-info=size "file://$(pwd)" "$object_id" >actual && + test_cmp expect actual + ) +' + +test_done diff --git a/transport-helper.c b/transport-helper.c index a0297b0986..64c175e904 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -686,13 +686,17 @@ static int fetch_refs(struct transport *transport, /* * If we reach here, then the server, the client, and/or the transport - * helper does not support protocol v2. --negotiate-only requires - * protocol v2. + * helper does not support protocol v2. --negotiate-only and --object-info + * require protocol v2. */ if (data->transport_options.acked_commits) { warning(_("--negotiate-only requires protocol v2")); return -1; } + if (transport->smart_options->object_info) { + warning(_("--object-info requires protocol v2")); + return -1; + } if (!data->get_refs_list_called) get_refs_list_using_list(transport, 0); diff --git a/transport-internal.h b/transport-internal.h index c4ca0b733a..04fa015011 100644 --- a/transport-internal.h +++ b/transport-internal.h @@ -59,6 +59,7 @@ struct transport_vtable { * use. disconnect() releases these resources. **/ int (*disconnect)(struct transport *connection); + int (*fetch_object_info)(struct transport *transport, struct oid_array *oids); }; #endif diff --git a/transport.c b/transport.c index 70e9840a90..65a1b1fdb3 100644 --- a/transport.c +++ b/transport.c @@ -350,6 +350,67 @@ static struct ref *handshake(struct transport *transport, int for_push, return refs; } +/* + * Fetches object info if server supports object-info command + * Make sure to fallback to normal fetch if this fails + */ +static int fetch_object_info(struct transport *transport) +{ + struct git_transport_data *data = transport->data; + struct object_info_args args; + struct packet_reader reader; + + memset(&args, 0, sizeof(args)); + args.server_options = transport->server_options; + args.object_info_options = transport->smart_options->object_info_options; + args.oids = transport->smart_options->object_info_oids; + + connect_setup(transport, 0); + packet_reader_init(&reader, data->fd[0], NULL, 0, + PACKET_READ_CHOMP_NEWLINE | + PACKET_READ_DIE_ON_ERR_PACKET); + data->version = discover_version(&reader); + + transport->hash_algo = reader.hash_algo; + + switch (data->version) { + case protocol_v2: + if (!server_supports_v2("object-info", 0)) + return 0; + send_object_info_request(data->fd[1], &args); + break; + case protocol_v1: + case protocol_v0: + die(_("wrong protocol version. expected v2")); + case protocol_unknown_version: + BUG("unknown protocol version"); + } + + if (packet_reader_read(&reader) != PACKET_READ_NORMAL) { + die(_("error reading object info header")); + } + if (strcmp(reader.line, "size")) { + die(_("expected 'size', received '%s'"), reader.line); + } + while (packet_reader_read(&reader) == PACKET_READ_NORMAL) { + printf("%s\n", reader.line); + } + check_stateless_delimiter(transport->stateless_rpc, &reader, "stateless delimiter expected"); + + return 1; +} + +static int append_oid_to_ref(const struct object_id *oid, void *data) +{ + struct ref *ref = data; + struct ref *temp_ref = xcalloc(1, sizeof (struct ref)); + temp_ref->old_oid = *oid; + temp_ref->exact_oid = 1; + ref->next = temp_ref; + ref = ref->next; + return 0; +} + static struct ref *get_refs_via_connect(struct transport *transport, int for_push, struct transport_ls_refs_options *options) { @@ -364,6 +425,7 @@ static int fetch_refs_via_pack(struct transport *transport, struct ref *refs = NULL; struct fetch_pack_args args; struct ref *refs_tmp = NULL; + struct ref *wanted_refs = xcalloc(1, sizeof (struct ref)); memset(&args, 0, sizeof(args)); args.uploadpack = data->options.uploadpack; @@ -388,8 +450,17 @@ static int fetch_refs_via_pack(struct transport *transport, args.server_options = transport->server_options; args.negotiation_tips = data->options.negotiation_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow; + args.object_info = transport->smart_options->object_info; - if (!data->got_remote_heads) { + if (transport->smart_options && transport->smart_options->object_info) { + struct ref *temp_ref = wanted_refs; + if (fetch_object_info(transport)) { + goto cleanup; + } + oid_array_for_each(transport->smart_options->object_info_oids, append_oid_to_ref, temp_ref); + wanted_refs = wanted_refs->next; + transport->remote_refs = wanted_refs; + } else if (!data->got_remote_heads) { int i; int must_list_refs = 0; for (i = 0; i < nr_heads; i++) { @@ -406,7 +477,7 @@ static int fetch_refs_via_pack(struct transport *transport, else if (data->version <= protocol_v1) die_if_server_options(transport); - if (data->options.acked_commits) { + if (data->options.acked_commits && !transport->smart_options->object_info) { if (data->version < protocol_v2) { warning(_("--negotiate-only requires protocol v2")); ret = -1; diff --git a/transport.h b/transport.h index a0bc6a1e9e..3bba16f86e 100644 --- a/transport.h +++ b/transport.h @@ -30,6 +30,12 @@ struct git_transport_options { */ unsigned connectivity_checked:1; + /* + * Transport will attempt to pull only object-info. Fallbacks + * to pulling entire object if object-info is not supported + */ + unsigned object_info : 1; + int depth; const char *deepen_since; const struct string_list *deepen_not; @@ -53,6 +59,9 @@ struct git_transport_options { * common commits to this oidset instead of fetching any packfiles. */ struct oidset *acked_commits; + + struct oid_array *object_info_oids; + struct string_list *object_info_options; }; enum transport_family {