From patchwork Fri Jun 28 19:04:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Ju X-Patchwork-Id: 13716534 Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com [209.85.160.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53B295381B for ; Fri, 28 Jun 2024 19:05:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719601543; cv=none; b=dYwUsT3qyXjU6QxXr0IJvcQRtZMFtRmHM9767wEf02db0+vdWYi4lA9IjyujCnaDn2Op5cHkuKdPsc8os9m18vwjpFZCpm8WjvS06i6hnsi0+9PtfPIa5MLDVEvQyhb7SHI7BTOoUVkocHcMzXMsNwNjsCnAxIx6hAQREWQWrL0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719601543; c=relaxed/simple; bh=Yj+KtyTn+ETBJEyD8XUiTtzBPoLHsC+5WdG+8Ug6UU0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=b9+i06y4Dv7o1kDs8XvhNTaYiJGMIBsMYxDNZ52kjBalqGR8KCiDfX+jwUmuiMEKTHKpenWwcRQzQS17BRAETd/Ql7xTeNGnUUhpozA2F9RfaFYOvCssJnu1dY+2JZW0mfihfXrcTiJPN8zqYjkqYjU/NxWFp8QcU/5QFmnnZLo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=SOB2DQdm; arc=none smtp.client-ip=209.85.160.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SOB2DQdm" Received: by mail-qt1-f172.google.com with SMTP id d75a77b69052e-44639c3d8e7so6030821cf.1 for ; Fri, 28 Jun 2024 12:05:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719601541; x=1720206341; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=URyEaRISo94s9l2pD7DVNc45D4IRWHLO9XQk42QL7BY=; b=SOB2DQdmI0eOrV2nbqu74HL8ugfqA7BRAt/iJItd5s9wYDZfq9OSBRI4h8OQ9uPEc5 YQZ8h5QGd/pU4iu23TuiMStSm7A2eB43c6E2gwXpZ6eiw/KgRUi3gyFGMyVa1d74Yofw N/E4YbN+df+tqMnWSYPcuborTSvVv64Fn3XKveuQwUtd4lmdPTSm5umulz1LPzyRKXLK 1RHzH27T9aXtC5noDTrSM1MglYlhxv3B9I+JFofKLvpUPZi2GPvQl+fJ2bLB+/x8T3U3 Jw8lLLp1uyJ7CXvsGjUKvDr8qKpkorg2r8T7EecxgBLo4uxnl3dOqXLj0Y5ThXId73D/ HeXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719601541; x=1720206341; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=URyEaRISo94s9l2pD7DVNc45D4IRWHLO9XQk42QL7BY=; b=qCbDwcSwx53eqdSu/P1geI3+mKpzweJAVuPcNGda5e+qdNW0pEQCxilHdd1lVSRmC9 mc+hR0he3Ie6fZrk8cLGnBMfKQRUrGKE1ESd8lJ7rvGZL8wYDD4xlkyoSIVS0sbLjPg5 EKMBhK0YBFQmDGUT83uzPccKqDK1wxeSDzVhtU6RnKgGNkt/xqz1WMVdOYLDo65BKgei JKfoFOBT2inY9+tJiaAOc9nsinElKFR6ucAnhIYSB4hIzTWsB3j3r6jytyCFY5p/GXdJ FSowylP/6QCKLdFJeVIL5kRCyZHkOqv5CfoTicqb8BKf8yPX2h5KfdrJwd0hwyhocIJa OItw== X-Gm-Message-State: AOJu0Yzn8jUgf9uKeGguc2zcbeYW232JulfHCK2cHuEMNd001R9E6dTj Wd2BPVfQddzs2SJcw6NBv8LASShdITOxRgeW39PBnFgOrqr9UE5dXzniLekt X-Google-Smtp-Source: AGHT+IF1d5v0hDRjYtpQHDjbwa4+cvs2c++RTxqMPMbIUmKeWyQlq8EWnn0svSQP7AMidVmlrBaRgg== X-Received: by 2002:ad4:4088:0:b0:6b0:7b41:ca94 with SMTP id 6a1803df08f44-6b53bcdab53mr177058446d6.33.1719601540848; Fri, 28 Jun 2024 12:05:40 -0700 (PDT) Received: from localhost.localdomain ([76.71.94.205]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b59e574cc4sm10262346d6.53.2024.06.28.12.05.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 28 Jun 2024 12:05:40 -0700 (PDT) From: Eric Ju To: git@vger.kernel.org Cc: Christian Couder , Calvin Wan , Jonathan Tan , John Cai , Eric Ju Subject: [PATCH 1/6] fetch-pack: refactor packet writing Date: Fri, 28 Jun 2024 15:04:58 -0400 Message-ID: <20240628190503.67389-2-eric.peijian@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240628190503.67389-1-eric.peijian@gmail.com> References: <20240628190503.67389-1-eric.peijian@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Calvin Wan A subsequent patch need to write capabilities for another command. Refactor write_fetch_command_and_capabilities() to be used by both fetch and future command. Signed-off-by: Calvin Wan Signed-off-by: Eric Ju Helped-by: Jonathan Tan Helped-by: Christian Couder --- fetch-pack.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index eba9e420ea..fc9fb66cd8 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1313,13 +1313,13 @@ static int add_haves(struct fetch_negotiator *negotiator, return haves_added; } -static void write_fetch_command_and_capabilities(struct strbuf *req_buf, - const struct string_list *server_options) +static void write_command_and_capabilities(struct strbuf *req_buf, + const struct string_list *server_options, const char* command) { const char *hash_name; - ensure_server_supports_v2("fetch"); - packet_buf_write(req_buf, "command=fetch"); + ensure_server_supports_v2(command); + packet_buf_write(req_buf, "command=%s", command); if (server_supports_v2("agent")) packet_buf_write(req_buf, "agent=%s", git_user_agent_sanitized()); if (advertise_sid && server_supports_v2("session-id")) @@ -1355,7 +1355,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, int done_sent = 0; struct strbuf req_buf = STRBUF_INIT; - write_fetch_command_and_capabilities(&req_buf, args->server_options); + write_command_and_capabilities(&req_buf, args->server_options, "fetch"); if (args->use_thin_pack) packet_buf_write(&req_buf, "thin-pack"); @@ -2163,7 +2163,7 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips, the_repository, "%d", negotiation_round); strbuf_reset(&req_buf); - write_fetch_command_and_capabilities(&req_buf, server_options); + write_command_and_capabilities(&req_buf, server_options, "fetch"); packet_buf_write(&req_buf, "wait-for-done"); From patchwork Fri Jun 28 19:04:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Ju X-Patchwork-Id: 13716535 Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 14CAC54662 for ; Fri, 28 Jun 2024 19:05:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719601544; cv=none; b=NTrP22YjzCCiwYmT27UMtEESS3pIipQ1/vB9aDvysibFSrb35PDWWTfbbmhoTm+zFw05BO1Qoh6+twQv4BT/JEo7CfAtssTWdDpZdgLPPjlN+OyiCHAwljbpelSaAslcND0qDaEbNH8fLYE43fK57kFVaKsuQV722ZL2cZc7aAQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719601544; c=relaxed/simple; bh=v5wyCHQBuLCg3n1RvRotf2Bds2mIKPfLnXxXAkv4vVM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=htmg+MSYszOrb6wGcmWVe9mF2086amOM7ahQJlLLOSx8XtG0e0xQItMo3jw0jMXTOitTPGvKMBuphqN3NuPDV4RKRRmbr0bL9K368FgtpiaHEEV++Yyo/wJd+TVWOHrdwCjJmrfEqw81PZXN9/EJDLCna111GB4Zigle0vcy17o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=a+SJCliq; arc=none smtp.client-ip=209.85.160.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="a+SJCliq" Received: by mail-qt1-f173.google.com with SMTP id d75a77b69052e-4449beccc7fso5527241cf.0 for ; Fri, 28 Jun 2024 12:05:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719601542; x=1720206342; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UeYBkhNyTT/Qoi6Sa0KM8qxv2snYCjXxbxnvtVkZMqk=; b=a+SJCliqooS/7Jz0MFuEgN2QZgqKFNEos/LuuJJD5deF/YSR5+X5AhBHBw4Hs+12Hr 4Hk3vpj1Vrl7kRlndxG+BHdRWmIjH9foyF6eowHEFrIQy+8csPi06xeQEGja3UsYpiTD FX053UoJjO0arFolTAVjgUea1OTt0dmUEbvshuNI8kfnBzg+kEJpRHtecc7riu+7Xv0V f1BBfPEfER1zCevGuwcryZArGW7x0nFyshhhkaUmi/t8hxAtSTaVx3BPfba6wBhmziyG KJ264FGJP3yjuOxGqyL/TN4p5h9IJbstMepXSJtTKydZVY+JSBKVHTxvNqAUBObgNIkF goeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719601542; x=1720206342; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UeYBkhNyTT/Qoi6Sa0KM8qxv2snYCjXxbxnvtVkZMqk=; b=QiS4RGUiBv9Xukoef06JvccNfi6BekvFPFoMWGKX9R8uN51yPQ9a9bsljRuunZRwFB 0skZA4R4aevWpeq5gvsV1QMXfz0G5UCfwm0LCA4HFDKBhF4mQhJm0yscKKSj3HZmk0jf chRWCqGoaTBEGdHltUDganmsUV5EcdCesPAPd4f4H3W1H6Ntx3ztq+ehHMCmU9r1G+5x 7se6p0NJ0OgZggWWCVbd6fSFopUHtJPUXjdVAI+1MUmjgkqiwp/xvKizwVCoo7IBgDgE oVlTKVEPmkM1il1/uoMhjlmb3oGI/GoiPtneiZKS04/LZ+g5/iZhn7z5+a7g7ZVcIHQH hzRQ== X-Gm-Message-State: AOJu0Yz/9eP0fUyLJ1KTId+7mR+TEXth14itsdjZaBRglowmNkpHem2B mur0i+PEJXroE+hEss47ZE+rOkz7XHZ7uWcVXA1HnxEztepsDTIqa4/yN/Wt X-Google-Smtp-Source: AGHT+IF3gzkB8jiThcCASfzX7SbjCXiINZAJ+80CutYnePwbXHmwse+F3FAmfriwx9Aj9qZ2XnqHrw== X-Received: by 2002:a05:6214:20e2:b0:6b5:2b33:5445 with SMTP id 6a1803df08f44-6b5409d9263mr247308286d6.25.1719601541725; Fri, 28 Jun 2024 12:05:41 -0700 (PDT) Received: from localhost.localdomain ([76.71.94.205]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b59e574cc4sm10262346d6.53.2024.06.28.12.05.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 28 Jun 2024 12:05:41 -0700 (PDT) From: Eric Ju To: git@vger.kernel.org Cc: Christian Couder , Calvin Wan , Jonathan Tan , John Cai , Eric Ju Subject: [PATCH 2/6] fetch-pack: move fetch initialization Date: Fri, 28 Jun 2024 15:04:59 -0400 Message-ID: <20240628190503.67389-3-eric.peijian@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240628190503.67389-1-eric.peijian@gmail.com> References: <20240628190503.67389-1-eric.peijian@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Calvin Wan There are some variables initialized at the start of the do_fetch_pack_v2() state machine. Currently, they are initialized in FETCH_CHECK_LOCAL, which is the initial state set at the beginning of the function. However, a subsequent patch will allow for another initial state, while still requiring these initialized variables. Move the initialization to be before the state machine, so that they are set regardless of the initial state. Note that there is no change in behavior, because we're moving code from the beginning of the first state to just before the execution of the state machine. Signed-off-by: Calvin Wan Signed-off-by: Eric Ju Helped-by: Jonathan Tan Helped-by: Christian Couder --- fetch-pack.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index fc9fb66cd8..da0de9c537 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1676,18 +1676,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, reader.me = "fetch-pack"; } + /* v2 supports these by default */ + allow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1; + use_sideband = 2; + if (args->depth > 0 || args->deepen_since || args->deepen_not) + args->deepen = 1; + while (state != FETCH_DONE) { switch (state) { case FETCH_CHECK_LOCAL: sort_ref_list(&ref, ref_compare_name); QSORT(sought, nr_sought, cmp_ref_by_name); - /* v2 supports these by default */ - allow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1; - use_sideband = 2; - if (args->depth > 0 || args->deepen_since || args->deepen_not) - args->deepen = 1; - /* Filter 'ref' by 'sought' and those that aren't local */ mark_complete_and_common_ref(negotiator, args, &ref); filter_refs(args, &ref, sought, nr_sought); From patchwork Fri Jun 28 19:05:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Ju X-Patchwork-Id: 13716536 Received: from mail-qv1-f44.google.com (mail-qv1-f44.google.com [209.85.219.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EEF2A6F2E8 for ; Fri, 28 Jun 2024 19:05:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719601545; cv=none; b=FWxAS/QvbGwRkQLQ2MveAq9gY94jF6VQhelqT/HpTRV4noaWt2PYXMW/5c//98aEI0Jps8X3acFtl/WdUq7L9dKBa/l/gpy3xo7gsyCGSbcjXmN302TSS+La8OMDxusCN2AvM0A5+Qe74vMvQmkMfWoYBWCX5dnH199c1Vd/zHs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719601545; c=relaxed/simple; bh=O6VsDUTpWVpNVuU2q5wW6JpdtaOCprl0QqvqIb07puw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=S1xEU0YQ7AamH4uPS42Bo/Mpen1f1UT4xztjCqR12QKAUTUGsPZHz1J8AkLaH/C7SXRasORlP2/fIqILpoRH1ivkVQMS8G3/jbB4MRXZKj3dTvEMQJEb9+WeQHYknd9Qh3k7eqc/bv1jW4ZHNjt0psqL/y6+ZYsyxxJ/UTW0hp0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=W5a/569r; arc=none smtp.client-ip=209.85.219.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="W5a/569r" Received: by mail-qv1-f44.google.com with SMTP id 6a1803df08f44-6b4ffc2a7abso7521776d6.1 for ; Fri, 28 Jun 2024 12:05:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719601542; x=1720206342; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=X2KD7IszvYFQRGgodF0M1Bc6eIEDIxHrXH2LkimTVYw=; b=W5a/569rziBCcFqMPEikpkeq8pa8WoqvhCQEpKkym6yuFZ4sJM7TqAuxmSOMrnVtLN wfMrx84fxBPuOPzK5016biuWu6W1JFiXtjBOAwGDU5bjjzIJo2fcDctFHNsfRh6cB0EH 1KRLFQwsLNEDB22FL3Zh62IGZNVexeMZAPUn3UQYLd5lGEjpHvJFJ2ZUa8NaXV6gD3n9 QY/NwxlXF9RZ6nVIc+dwZejW+ARaWad3SR7frRGAUP/tY8Ru9Viomq2f+JAyJ88jLhgS 6jUwVTy1YAkr+/n74jPx7410IQNGcCSa3UvbVo6cnVp3zQo/CIgzwbvRB6YCZTWc3Zwp eHFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719601542; x=1720206342; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X2KD7IszvYFQRGgodF0M1Bc6eIEDIxHrXH2LkimTVYw=; b=J2M5UQO9foTbjPoiFL469LIoGkMrZ+6MgqyzvDj41GwSx8tEoa1vgc7s15hDQHLnPE ujhYobFrKhg4c3GdjUnx5gLx1xqCIAm2VnAkSO2Gu5w5h+BG0oAl+gdzMU9RiROepoom adQkNxPeF+nB6oYL+CVswCisgUtxSQVqUSWXxY4GLPivLJlkqUWzngHGu7+b2iilOVEt Jh9U60t2Dv0cDvJONJZ+yJP/MOLUzMMx2qWsVmCPmri2hqe8ewTz9JswBwNvZhboH99U +uAiUXiDVk8vczpQZ2bN4qyPN72mkAUI6J1Ks3TtoeKqgoRC9rpkIF3Sqiw1Gp3l7y/t RF6g== X-Gm-Message-State: AOJu0YzmYk/M/OhWWv1VmboKwaVwyicH0DbW6nAkW7b9dZrs4S94gq8s +x6aB8xIVbQGNtPOhpX4kXHzeZ/AATT8YJu3MdpWXhyXBgs/QEWILBmemS5t X-Google-Smtp-Source: AGHT+IHL3LZ6RwrbtHVsWJRPMgxC0ke+2cFzAyFmp5PZl/9ixBCDW2pqNmYfeKjCiT2pjMSIAa1nKg== X-Received: by 2002:a05:6214:27ef:b0:6b0:8991:a2f7 with SMTP id 6a1803df08f44-6b5a541c10dmr34354196d6.12.1719601542491; Fri, 28 Jun 2024 12:05:42 -0700 (PDT) Received: from localhost.localdomain ([76.71.94.205]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b59e574cc4sm10262346d6.53.2024.06.28.12.05.41 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 28 Jun 2024 12:05:42 -0700 (PDT) From: Eric Ju To: git@vger.kernel.org Cc: Christian Couder , Calvin Wan , Jonathan Tan , John Cai , Eric Ju Subject: [PATCH 3/6] serve: advertise object-info feature Date: Fri, 28 Jun 2024 15:05:00 -0400 Message-ID: <20240628190503.67389-4-eric.peijian@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240628190503.67389-1-eric.peijian@gmail.com> References: <20240628190503.67389-1-eric.peijian@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Calvin Wan In order for a client to know what object-info components a server can provide, advertise supported object-info features. This will allow a client to decide whether to query the server for object-info or fetch as a fallback. Signed-off-by: Calvin Wan Signed-off-by: Eric Ju Helped-by: Jonathan Tan Helped-by: Christian Couder --- serve.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/serve.c b/serve.c index aa651b73e9..fd42fecc15 100644 --- a/serve.c +++ b/serve.c @@ -68,7 +68,7 @@ static void session_id_receive(struct repository *r UNUSED, trace2_data_string("transfer", NULL, "client-sid", client_sid); } -static int object_info_advertise(struct repository *r, struct strbuf *value UNUSED) +static int object_info_advertise(struct repository *r, struct strbuf *value) { if (advertise_object_info == -1 && repo_config_get_bool(r, "transfer.advertiseobjectinfo", @@ -76,6 +76,8 @@ static int object_info_advertise(struct repository *r, struct strbuf *value UNUS /* disabled by default */ advertise_object_info = 0; } + if (value && advertise_object_info) + strbuf_addstr(value, "size"); return advertise_object_info; } From patchwork Fri Jun 28 19:05:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eric Ju X-Patchwork-Id: 13716537 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A52B96F31D for ; Fri, 28 Jun 2024 19:05:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719601546; cv=none; b=Y2UeejmKe7X+Cw/eU6hMw8Xbh6MdSnMDnuUy+mmIDmnyj86e7qeSw7ijOsblWgQmcU5+8TyAXMB+GDwEsMcpv6EkyOr0SeMC0RxCs1+TDJPUKngS+da9rqcblA2Brq/msQlrlmpkdPOEdPkrx+FRalscJZ7U5keDxMjg5QY/2Ww= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719601546; c=relaxed/simple; bh=0EXRJqOOTw8Ad+PqEMaRowif7YgUj9RiyVTs8qglbVw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=jWFjGc8tOyZKYgsGhP028koivQzoVqg63MAoV4uQnJvZ8qcaXAT3NBC4wWgW4Lgnyyt7Cdk+ruCbvn/zeGW+zoNSgolxXutjvoIvKaUWtKqSGYs55XRjhbySAaaxyNZtrj+4FhcdE0ruFza7qLh4Y49WfwYvSDzBBgsqPGJvuXI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lY6+Mnoo; arc=none smtp.client-ip=209.85.222.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lY6+Mnoo" Received: by mail-qk1-f177.google.com with SMTP id af79cd13be357-79c03abfb18so57108785a.0 for ; Fri, 28 Jun 2024 12:05:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719601543; x=1720206343; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8Z85AKYgRFs3MopR+pp6xiAO7JDy2XkLAJdsJ8h7Qh8=; b=lY6+Mnoo9IgJWjJMIjBWzpO9d1CJWbpENd1jjjz5Eo4adHYAXSCyRDul3yyj5RCSRt 4VGhOVPeZzgaAYzIpjbdgdiATTcaaXqhJoFFbcrrPxJMNisjJFSa0LkOzlq3pAKC0pn/ cDdmwW4B9M0htUzzxCMsxUzQX+ZIHaDYBn7kTGpv017EBMj51sFBf5Wen3xUDMa+v+v/ TVmVJUzO1dlH8rHZA655YLLPemY1tNv/1pqFrbTe/xjFlqboVX9XkvSwR22HlzxJmPI0 WJ9+io+ZlDzWFzrPF+QETWvW6OYYgfbPWvmdwedsFG/MhkBcjXLxEbbZ9oPqov0zdWZC oPmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719601543; x=1720206343; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8Z85AKYgRFs3MopR+pp6xiAO7JDy2XkLAJdsJ8h7Qh8=; b=BjxaQNq0DXPRlqiJNXUNRCOGgFRkyF+kKUr+T/01vTi/RkP+HyXZUBlRErhPYGrUm3 Qu4UdwIZDWvJWZ1VYOBnj6eFLDdHSuWvvujQe9ARkGZurzraU/8eWcRsc5hIA1lUGC+5 cQFgxT34cnX+0XGlYBs4rcOa25ZRy5qyFPvceASGPn0XcfZEtQsG0IShLfhhDzb4uGkX 37eu2kLCZpBAszcYkDgyr4oQoORVDo/un9buzBEs14lwZ9YaiAjTstmOi++xLiI3DbDH BUfQ1oP+ZYoiv8k2PgQ4WWwdwIyJ58e8kAZNcmA+bfLBOa/tcXOdSabTkLm0pgIUmpQ7 ALvg== X-Gm-Message-State: AOJu0YxiDpRbKP0kVkUvtWtQ3ZNoreYvlKXBbrtyD/4eGvu3/G+Rio9j iBHv252mCwZBFN7xLLtLdXtzqxwqWkwkqQuLHNYoaB1CXotgMwqbmCIQtzoI X-Google-Smtp-Source: AGHT+IFQwKWC8s5xmoBywQijyMIC/I/FHjmzf0sDCckKnbQRiQc9Owk8aSNWX1LOgliGHwsT25W6KQ== X-Received: by 2002:a05:6214:c64:b0:6b5:ab9d:d273 with SMTP id 6a1803df08f44-6b5ab9dd357mr18772396d6.1.1719601543311; Fri, 28 Jun 2024 12:05:43 -0700 (PDT) Received: from localhost.localdomain ([76.71.94.205]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b59e574cc4sm10262346d6.53.2024.06.28.12.05.42 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 28 Jun 2024 12:05:42 -0700 (PDT) From: Eric Ju To: git@vger.kernel.org Cc: Christian Couder , Calvin Wan , Jonathan Tan , John Cai , Eric Ju Subject: [PATCH 4/6] transport: add client support for object-info Date: Fri, 28 Jun 2024 15:05:01 -0400 Message-ID: <20240628190503.67389-5-eric.peijian@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240628190503.67389-1-eric.peijian@gmail.com> References: <20240628190503.67389-1-eric.peijian@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Calvin Wan Sometimes it is useful to get information about an object without having to download it completely. The server logic has already been implemented as “a2ba162cda (object-info: support for retrieving object info, 2021-04-20)”. Add client functions to communicate with the server. The client currently supports requesting a list of object ids with features 'size' and 'type' from a v2 server. If a server does not advertise either of the requested features, then the client falls back to making the request through 'fetch'. Signed-off-by: Calvin Wan Signed-off-by: Eric Ju Helped-by: Jonathan Tan Helped-by: Christian Couder --- fetch-pack.c | 24 +++++++++++ fetch-pack.h | 10 +++++ transport-helper.c | 8 +++- transport.c | 102 ++++++++++++++++++++++++++++++++++++++++++--- transport.h | 11 +++++ 5 files changed, 148 insertions(+), 7 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index da0de9c537..d533cac1d8 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1345,6 +1345,27 @@ static void write_command_and_capabilities(struct strbuf *req_buf, packet_buf_delim(req_buf); } +void send_object_info_request(int fd_out, struct object_info_args *args) +{ + struct strbuf req_buf = STRBUF_INIT; + + write_command_and_capabilities(&req_buf, args->server_options, "object-info"); + + if (unsorted_string_list_has_string(args->object_info_options, "size")) + packet_buf_write(&req_buf, "size"); + + if (args->oids) { + for (size_t i = 0; i < args->oids->nr; i++) + packet_buf_write(&req_buf, "oid %s", oid_to_hex(&args->oids->oid[i])); + } + + packet_buf_flush(&req_buf); + if (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0) + die_errno(_("unable to write request to remote")); + + strbuf_release(&req_buf); +} + static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, struct fetch_pack_args *args, const struct ref *wants, struct oidset *common, @@ -1682,6 +1703,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, if (args->depth > 0 || args->deepen_since || args->deepen_not) args->deepen = 1; + if (args->object_info) + state = FETCH_SEND_REQUEST; + while (state != FETCH_DONE) { switch (state) { case FETCH_CHECK_LOCAL: diff --git a/fetch-pack.h b/fetch-pack.h index 6775d26517..16e4dc0824 100644 --- a/fetch-pack.h +++ b/fetch-pack.h @@ -16,6 +16,7 @@ struct fetch_pack_args { const struct string_list *deepen_not; struct list_objects_filter_options filter_options; const struct string_list *server_options; + struct object_info **object_info_data; /* * If not NULL, during packfile negotiation, fetch-pack will send "have" @@ -42,6 +43,7 @@ struct fetch_pack_args { unsigned reject_shallow_remote:1; unsigned deepen:1; unsigned refetch:1; + unsigned object_info:1; /* * Indicate that the remote of this request is a promisor remote. The @@ -68,6 +70,12 @@ struct fetch_pack_args { unsigned connectivity_checked:1; }; +struct object_info_args { + struct string_list *object_info_options; + const struct string_list *server_options; + struct oid_array *oids; +}; + /* * sought represents remote references that should be updated from. * On return, the names that were found on the remote will have been @@ -101,4 +109,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips, */ int report_unmatched_refs(struct ref **sought, int nr_sought); +void send_object_info_request(int fd_out, struct object_info_args *args); + #endif diff --git a/transport-helper.c b/transport-helper.c index 9820947ab2..670d1e7068 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -697,13 +697,17 @@ static int fetch_refs(struct transport *transport, /* * If we reach here, then the server, the client, and/or the transport - * helper does not support protocol v2. --negotiate-only requires - * protocol v2. + * helper does not support protocol v2. --negotiate-only and cat-file remote-object-info + * require protocol v2. */ if (data->transport_options.acked_commits) { warning(_("--negotiate-only requires protocol v2")); return -1; } + if (transport->smart_options->object_info) { + // fail the command explicitly to avoid further commands input + die(_("remote-object-info requires protocol v2")); + } if (!data->get_refs_list_called) get_refs_list_using_list(transport, 0); diff --git a/transport.c b/transport.c index 83ddea8fbc..2847aa3f3c 100644 --- a/transport.c +++ b/transport.c @@ -363,6 +363,73 @@ static struct ref *handshake(struct transport *transport, int for_push, return refs; } +static int fetch_object_info(struct transport *transport, struct object_info **object_info_data) +{ + int size_index = -1; + struct git_transport_data *data = transport->data; + struct object_info_args args; + struct packet_reader reader; + + memset(&args, 0, sizeof(args)); + args.server_options = transport->server_options; + args.object_info_options = transport->smart_options->object_info_options; + args.oids = transport->smart_options->object_info_oids; + + connect_setup(transport, 0); + packet_reader_init(&reader, data->fd[0], NULL, 0, + PACKET_READ_CHOMP_NEWLINE | + PACKET_READ_GENTLE_ON_EOF | + PACKET_READ_DIE_ON_ERR_PACKET); + data->version = discover_version(&reader); + + transport->hash_algo = reader.hash_algo; + + switch (data->version) { + case protocol_v2: + if (!server_supports_v2("object-info")) + return -1; + if (unsorted_string_list_has_string(args.object_info_options, "size") + && !server_supports_feature("object-info", "size", 0)) { + return -1; + } + send_object_info_request(data->fd[1], &args); + break; + case protocol_v1: + case protocol_v0: + die(_("wrong protocol version. expected v2")); + case protocol_unknown_version: + BUG("unknown protocol version"); + } + + for (size_t i = 0; i < args.object_info_options->nr; i++) { + if (packet_reader_read(&reader) != PACKET_READ_NORMAL) { + check_stateless_delimiter(transport->stateless_rpc, &reader, "stateless delimiter expected"); + return -1; + } + if (unsorted_string_list_has_string(args.object_info_options, reader.line)) { + if (!strcmp(reader.line, "size")) + size_index = i; + continue; + } + return -1; + } + + for (size_t i = 0; packet_reader_read(&reader) == PACKET_READ_NORMAL && i < args.oids->nr; i++){ + struct string_list object_info_values = STRING_LIST_INIT_DUP; + + string_list_split(&object_info_values, reader.line, ' ', -1); + if (0 <= size_index) { + if (!strcmp(object_info_values.items[1 + size_index].string, "")) + die("object-info: not our ref %s", + object_info_values.items[0].string); + *(*object_info_data)[i].sizep = strtoul(object_info_values.items[1 + size_index].string, NULL, 10); + } + } + check_stateless_delimiter(transport->stateless_rpc, &reader, "stateless delimiter expected"); + + return 0; +} + static struct ref *get_refs_via_connect(struct transport *transport, int for_push, struct transport_ls_refs_options *options) { @@ -410,6 +477,7 @@ static int fetch_refs_via_pack(struct transport *transport, struct ref *refs = NULL; struct fetch_pack_args args; struct ref *refs_tmp = NULL; + struct ref *object_info_refs = xcalloc(1, sizeof (struct ref)); memset(&args, 0, sizeof(args)); args.uploadpack = data->options.uploadpack; @@ -436,11 +504,27 @@ static int fetch_refs_via_pack(struct transport *transport, args.server_options = transport->server_options; args.negotiation_tips = data->options.negotiation_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow; - - if (!data->finished_handshake) { - int i; + args.object_info = transport->smart_options->object_info; + + if (transport->smart_options && transport->smart_options->object_info) { + struct ref *ref = object_info_refs; + + if (!fetch_object_info(transport, data->options.object_info_data)) + goto cleanup; + args.object_info_data = data->options.object_info_data; + args.quiet = 1; + args.no_progress = 1; + for (size_t i = 0; i < transport->smart_options->object_info_oids->nr; i++) { + struct ref *temp_ref = xcalloc(1, sizeof (struct ref)); + temp_ref->old_oid = *(transport->smart_options->object_info_oids->oid + i); + temp_ref->exact_oid = 1; + ref->next = temp_ref; + ref = ref->next; + } + transport->remote_refs = object_info_refs->next; + } else if (!data->finished_handshake) { int must_list_refs = 0; - for (i = 0; i < nr_heads; i++) { + for (int i = 0; i < nr_heads; i++) { if (!to_fetch[i]->exact_oid) { must_list_refs = 1; break; @@ -478,11 +562,18 @@ static int fetch_refs_via_pack(struct transport *transport, &transport->pack_lockfiles, data->version); data->finished_handshake = 0; + if (args.object_info) { + struct ref *ref_cpy_reader = object_info_refs->next; + for (int i = 0; ref_cpy_reader; i++) { + oid_object_info_extended(the_repository, &ref_cpy_reader->old_oid, &(*args.object_info_data)[i], OBJECT_INFO_LOOKUP_REPLACE); + ref_cpy_reader = ref_cpy_reader->next; + } + } data->options.self_contained_and_connected = args.self_contained_and_connected; data->options.connectivity_checked = args.connectivity_checked; - if (!refs) + if (!refs && !args.object_info) ret = -1; if (report_unmatched_refs(to_fetch, nr_heads)) ret = -1; @@ -498,6 +589,7 @@ static int fetch_refs_via_pack(struct transport *transport, free_refs(refs_tmp); free_refs(refs); list_objects_filter_release(&args.filter_options); + free_refs(object_info_refs); return ret; } diff --git a/transport.h b/transport.h index 6393cd9823..5a3cda1860 100644 --- a/transport.h +++ b/transport.h @@ -5,6 +5,7 @@ #include "remote.h" #include "list-objects-filter-options.h" #include "string-list.h" +#include "object-store.h" struct git_transport_options { unsigned thin : 1; @@ -30,6 +31,12 @@ struct git_transport_options { */ unsigned connectivity_checked:1; + /* + * Transport will attempt to pull only object-info. Fallbacks + * to pulling entire object if object-info is not supported. + */ + unsigned object_info : 1; + int depth; const char *deepen_since; const struct string_list *deepen_not; @@ -53,6 +60,10 @@ struct git_transport_options { * common commits to this oidset instead of fetching any packfiles. */ struct oidset *acked_commits; + + struct oid_array *object_info_oids; + struct object_info **object_info_data; + struct string_list *object_info_options; }; enum transport_family { From patchwork Fri Jun 28 19:05:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Ju X-Patchwork-Id: 13716538 Received: from mail-qv1-f49.google.com (mail-qv1-f49.google.com [209.85.219.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 772A47441A for ; Fri, 28 Jun 2024 19:05:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719601547; cv=none; b=ShPPVthBKJx4lUa4EgMIc+gLSR4Exw3ckDRWOgjjWyg9QP9gNQIzpRGiXAyT6GQAcgTiqNGmsV1n3ovTX8Eim+orUyHp7zmgaWrs+RazR4l9FcDjfBldjM6mOcM/aN8g1GZmZh7SLpSWonV5cWcyMwMz6XmX1uOemXMy9XkVhoc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719601547; c=relaxed/simple; bh=laTlMFAAmmBbP2uCpLm+OZgWTZBU3tZkzQRDYvj2Jeg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BsVwkTJzXGTjkgmIg50IZgI44e7AgVDyay0ymluIaPDYt2nj+chLi6how85MMpRFvLIN3tHYU900/gRnus8TQsZVCNveTeavLx494HV8GvDul+1eEeNQVkOXtUTv2Hg7jwxpI08fzC69hy0SOduZGS8pOYJXGSWo1dQh/0ke9Oo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Xa2uhimW; arc=none smtp.client-ip=209.85.219.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Xa2uhimW" Received: by mail-qv1-f49.google.com with SMTP id 6a1803df08f44-6ad86f3cc34so4896646d6.1 for ; Fri, 28 Jun 2024 12:05:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719601544; x=1720206344; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+slh/48oQBtspnNSvuscqc1tLioXexbTriSECz5EdVs=; b=Xa2uhimWhcXg1nljP6wgGWv844v3wPI2y7JQLrcdK5AoCyZM5G+M3JPwB6bCJteXXh Akzp9t8/sO2mz4lxW8QTaXTxinHQVyYRvzMg6GCHjxH56iz3kCMT6CwkRbklgavJiNXV wyGeTRvYUaTkowtz99MeYpyKzasWdLZ6HSgx8BqZ0CvR+xWl7if1x4yu26VW061M0vDb kWzRYtIl6JRjUeEkXKGGhJnloIOrqn/SfPCiOG2DqCdaWFXXhuViTr3WekQBOvTRQC/j +QS8jGY20Dp9F2iua6clLdACqSwhRd93uCitXWoRleYhXU5iy1A4AEUz7uGIpY8TdiAi ONUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719601544; x=1720206344; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+slh/48oQBtspnNSvuscqc1tLioXexbTriSECz5EdVs=; b=tOvzjZxHDo14Kne0CIitu0i+1H841LMoWbCp4sVN9A8z0+WTopmykYInO/6ED9n5lq Sr/fzcFUkRx1UdU2oAZHxJJODmARFqrF8XtolSgnD5iWjZTuJ5CtHaIdFmrXzsmgMKL2 t0H+r2VXgzM8yipdJoNAYGgq20UxSu46oTt22VQu/RZWJN13WuXemC7hKUNQd4MQLRks ZkZcszf2nfAOV3ZvLscYN2R4puvrRHLzmyxLYqkYJ0/0wqN8ehp5Px7T5s+Ydt5ehUfx 92ATafkh5JBWozafFqmo/Ztz6Hk/XYtlqw53oVrn120bfEksaQAe52RvD7QAhNro5ZNX +JDg== X-Gm-Message-State: AOJu0YyxHUsyLuCdh7nDZ9Y5kh5rDyXCA6SJO7qd+cL2GVooQmUQ9Mta osgV8LTpZkRPMTXoy7H3TbtSWZrXXzVAz2/9/kk9LqchKOxH8gOY1/pFkjBj X-Google-Smtp-Source: AGHT+IGLhknbow2p7+1OLZFOZ+a4qktRli0+8zO1QbTc3a1HW5Xckfd1sFHWQ8EM4z+QHJV1vlBGUw== X-Received: by 2002:a0c:ac05:0:b0:6b5:3da3:adaa with SMTP id 6a1803df08f44-6b53da3ae08mr170879596d6.3.1719601544058; Fri, 28 Jun 2024 12:05:44 -0700 (PDT) Received: from localhost.localdomain ([76.71.94.205]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b59e574cc4sm10262346d6.53.2024.06.28.12.05.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 28 Jun 2024 12:05:43 -0700 (PDT) From: Eric Ju To: git@vger.kernel.org Cc: Christian Couder , Calvin Wan , Jonathan Tan , John Cai , Eric Ju Subject: [PATCH 5/6] cat-file: add declaration of variable i inside its for loop Date: Fri, 28 Jun 2024 15:05:02 -0400 Message-ID: <20240628190503.67389-6-eric.peijian@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240628190503.67389-1-eric.peijian@gmail.com> References: <20240628190503.67389-1-eric.peijian@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Some code declares variable i and only uses it in a for loop, not in any other logic outside the loop. Change the declaration of i to be inside the for loop for readability. Signed-off-by: Eric Ju Helped-by: Christian Couder --- builtin/cat-file.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 43a1d7ac49..72a78cdc8c 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -668,12 +668,10 @@ static void dispatch_calls(struct batch_options *opt, struct queued_cmd *cmd, int nr) { - int i; - if (!opt->buffer_output) die(_("flush is only for --buffer mode")); - for (i = 0; i < nr; i++) + for (int i = 0; i < nr; i++) cmd[i].fn(opt, cmd[i].line, output, data); fflush(stdout); @@ -681,9 +679,7 @@ static void dispatch_calls(struct batch_options *opt, static void free_cmds(struct queued_cmd *cmd, size_t *nr) { - size_t i; - - for (i = 0; i < *nr; i++) + for (size_t i = 0; i < *nr; i++) FREE_AND_NULL(cmd[i].line); *nr = 0; @@ -709,7 +705,6 @@ static void batch_objects_command(struct batch_options *opt, size_t alloc = 0, nr = 0; while (strbuf_getdelim_strip_crlf(&input, stdin, opt->input_delim) != EOF) { - int i; const struct parse_cmd *cmd = NULL; const char *p = NULL, *cmd_end; struct queued_cmd call = {0}; @@ -719,7 +714,7 @@ static void batch_objects_command(struct batch_options *opt, if (isspace(*input.buf)) die(_("whitespace before command: '%s'"), input.buf); - for (i = 0; i < ARRAY_SIZE(commands); i++) { + for (int i = 0; i < ARRAY_SIZE(commands); i++) { if (!skip_prefix(input.buf, commands[i].name, &cmd_end)) continue; From patchwork Fri Jun 28 19:05:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Ju X-Patchwork-Id: 13716539 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CB4576048 for ; Fri, 28 Jun 2024 19:05:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719601549; cv=none; b=Q9wY4IFAzEEjGdoSdNEgOjD2wlfy0VMpfjtq8YWN0NcZhJ9RdhZeKHLf8cmopXxT/zNr/VWqXyc02A/ywWbGkR12SRKr+6HAuqFlMnZvdmXq4EIzjCCV0SOrtxzKWnkRDnlomdCgJaMUaynn2oBJGEJ4cTSHNf+A5oQhPtvZITY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719601549; c=relaxed/simple; bh=dcPGPMcnYSJGQ6TXU68AXVBaFDgn3eolIyjRV+vAB1M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YIRxCFX4TZ9iGsF6PwuTFAc/D148ehYMZ+jkUXBkek68ADeEEwPb9KE/u+V5E5FBIiLwu67iqMRdGIwjaUaAjHTgV2BPZXA+wXbI6Z9PqTBTBFemcIpIw5oo3QUUYSALHdmMyTf+SA/sFMB9yKkPt19T5HRJoqBl4zPx5o/c55Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JPwgOoUl; arc=none smtp.client-ip=209.85.219.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JPwgOoUl" Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-6b4ff803ef2so4985496d6.0 for ; Fri, 28 Jun 2024 12:05:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719601545; x=1720206345; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GhWo9w4Qz/L7e8C+MlRA+cl5G20WOx2Ihy5N2AQ42fc=; b=JPwgOoUlQbErynIfmaPQRzt1JMJTWErv4/MqZsf3wVv9/PL9YnUCML8YK7k88TlALC arInqHASFFApG/NZL4EIh4bxA/f5nuH9Twxupp2eq/uK6xOBymulSNXTI2XStzk+cNr/ 17fXA0zEjHRavu1q/Pv98gutX1QnMwqjPwVgvj4JunOMl5y0PVwhS18ex2NB0ATO95Fu wORAVBa7o6f+RwZHLBkXnHh5oAq91hEv9C2LzImREiptbBcFmbGs0f5lBUFq9ImGztg0 4kEkXElek0aeZ8UqMM7ubKxI3cWnIbDl1o/ZIJmNamdYDp1N2p8AaWJmoTHZQGUkpvk/ shRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719601545; x=1720206345; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GhWo9w4Qz/L7e8C+MlRA+cl5G20WOx2Ihy5N2AQ42fc=; b=ABOCd8puJrgAY/kE59vIZt+ywUd6kFQNCCEU9dv26DWCnZuU9aAL2/bgl5RILNM+CT X7U/K1l2sQC3kDfcBrQ0aTWr1BpXJ1HUKeyPFuVxuY75edi9xlrDyNob2Rv61jaiQYer 6RfCADVg8Yhxmavrm2thmiA3b6D9OhC6XhZyRPnGbkfcOQXmKt5zLFo+dEsIp/w138xt zEyIV0IncFo2e+cTY4lio1AUzQlaEKeFgvabRYDNTyXDwLxGLgr2kzNS/nHUiS3OEbxJ wwIIB3S596NgMRGTHRt4tRsG3XG+kVi9asbrLo2qIczCR2ZMbP/Rd9igqzYBbgFnU6xf moEA== X-Gm-Message-State: AOJu0YyguEb/L8CEAj0UqfS1tpdRLsiFj88hL8uZCHvUcwV24X1rfycu PK3meODvZ8WzbWQGBWsau5t9OPJoUM3F1liiyfwyqpYq+jHeritAppOSaEGm X-Google-Smtp-Source: AGHT+IEOJ9BEdIAewVz3wvenQ26B8H+z6ZqV3UkOAsn84LuNj9r+fvDwFvYJnv1xJSYfdLNDk3ANTw== X-Received: by 2002:a05:6214:20af:b0:6b5:a000:81ad with SMTP id 6a1803df08f44-6b5a00082bcmr42230366d6.63.1719601544991; Fri, 28 Jun 2024 12:05:44 -0700 (PDT) Received: from localhost.localdomain ([76.71.94.205]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b59e574cc4sm10262346d6.53.2024.06.28.12.05.44 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 28 Jun 2024 12:05:44 -0700 (PDT) From: Eric Ju To: git@vger.kernel.org Cc: Christian Couder , Calvin Wan , Jonathan Tan , John Cai , Eric Ju Subject: [PATCH 6/6] cat-file: add remote-object-info to batch-command Date: Fri, 28 Jun 2024 15:05:03 -0400 Message-ID: <20240628190503.67389-7-eric.peijian@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240628190503.67389-1-eric.peijian@gmail.com> References: <20240628190503.67389-1-eric.peijian@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Calvin Wan Since the `info` command in cat-file --batch-command prints object info for a given object, it is natural to add another command in cat-file --batch-command to print object info for a given object from a remote. Add `remote-object-info` to cat-file --batch-command. While `info` takes object ids one at a time, this creates overhead when making requests to a server so `remote-object-info` instead can take multiple object ids at once. cat-file --batch-command is generally implemented in the following manner: - Receive and parse input from user - Call respective function attached to command - Set batch mode state, get object info, print object info In --buffer mode, this changes to: - Receive and parse input from user - Store respective function attached to command in a queue - After flush, loop through commands in queue - Call respective function attached to command - Set batch mode state, get object info, print object info Notice how the getting and printing of object info is accomplished one at a time. As described above, this creates a problem for making requests to a server. Therefore, `remote-object-info` is implemented in the following manner: - Receive and parse input from user If command is `remote-object-info`: - Get object info from remote - Loop through object info - Call respective function attached to `info` - Set batch mode state, use passed in object info, print object info Else: - Call respective function attached to command - Parse input, get object info, print object info And finally for --buffer mode `remote-object-info`: - Receive and parse input from user - Store respective function attached to command in a queue - After flush, loop through commands in queue: If command is `remote-object-info`: - Get object info from remote - Loop through object info - Call respective function attached to `info` - Set batch mode state, use passed in object info, print object info Else: - Call respective function attached to command - Set batch mode state, get object info, print object info To summarize, `remote-object-info` gets object info from the remote and then generates multiple `info` commands with the object info passed in. In order for remote-object-info to avoid remote communication overhead in the non-buffer mode, the objects are passed in as such: remote-object-info ... rather than remote-object-info remote-object-info ... remote-object-info Signed-off-by: Calvin Wan Signed-off-by: Eric Ju Helped-by: Jonathan Tan Helped-by: Christian Couder --- Documentation/git-cat-file.txt | 22 +- builtin/cat-file.c | 231 ++++++++++---- object-file.c | 11 + object-store-ll.h | 3 + t/t1017-cat-file-remote-object-info.sh | 412 +++++++++++++++++++++++++ 5 files changed, 620 insertions(+), 59 deletions(-) create mode 100755 t/t1017-cat-file-remote-object-info.sh diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt index bd95a6c10a..ab0647bb39 100644 --- a/Documentation/git-cat-file.txt +++ b/Documentation/git-cat-file.txt @@ -149,6 +149,12 @@ info :: Print object info for object reference ``. This corresponds to the output of `--batch-check`. +remote-object-info ...:: + Print object info for object references `` at specified without + downloading object from remote. + Error when no object references is provided. + This command may be combined with `--buffer`. + flush:: Used with `--buffer` to execute all preceding commands that were issued since the beginning or since the last flush was issued. When `--buffer` @@ -290,7 +296,8 @@ newline. The available atoms are: The full hex representation of the object name. `objecttype`:: - The type of the object (the same as `cat-file -t` reports). + The type of the object (the same as `cat-file -t` reports). See + `CAVEATS` below. Not supported by `remote-object-info`. `objectsize`:: The size, in bytes, of the object (the same as `cat-file -s` @@ -298,13 +305,14 @@ newline. The available atoms are: `objectsize:disk`:: The size, in bytes, that the object takes up on disk. See the - note about on-disk sizes in the `CAVEATS` section below. + note about on-disk sizes in the `CAVEATS` section below. Not + supported by `remote-object-info`. `deltabase`:: If the object is stored as a delta on-disk, this expands to the full hex representation of the delta base object name. Otherwise, expands to the null OID (all zeroes). See `CAVEATS` - below. + below. Not supported by `remote-object-info`. `rest`:: If this atom is used in the output string, input lines are split @@ -314,7 +322,9 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname) -%(objecttype) %(objectsize)`. +%(objecttype) %(objectsize)`, except remote-object-info command who uses +`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet. +When "%(objecttype)" is supported, default format should be unified. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting @@ -396,6 +406,10 @@ scripting purposes. CAVEATS ------- +Note that since objecttype, objectsize:disk and deltabase are currently not supported by the +remote-object-info, git will error and exit when they are in the format string. + + Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are responsible for disk usage. The size of a packed non-delta object may be diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 72a78cdc8c..34958a1747 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -24,6 +24,9 @@ #include "promisor-remote.h" #include "mailmap.h" #include "write-or-die.h" +#include "alias.h" +#include "remote.h" +#include "transport.h" enum batch_mode { BATCH_MODE_CONTENTS, @@ -42,9 +45,14 @@ struct batch_options { char input_delim; char output_delim; const char *format; + int use_remote_info; }; +#define DEFAULT_FORMAT "%(objectname) %(objecttype) %(objectsize)" + static const char *force_path; +static struct object_info *remote_object_info; +static struct oid_array object_info_oids = OID_ARRAY_INIT; static struct string_list mailmap = STRING_LIST_INIT_NODUP; static int use_mailmap; @@ -508,7 +516,6 @@ static void batch_object_write(const char *obj_name, } batch_write(opt, scratch->buf, scratch->len); - if (opt->batch_mode == BATCH_MODE_CONTENTS) { print_object_or_die(opt, data); batch_write(opt, &opt->output_delim, 1); @@ -526,51 +533,118 @@ static void batch_one_object(const char *obj_name, (opt->follow_symlinks ? GET_OID_FOLLOW_SYMLINKS : 0); enum get_oid_result result; - result = get_oid_with_context(the_repository, obj_name, - flags, &data->oid, &ctx); - if (result != FOUND) { - switch (result) { - case MISSING_OBJECT: - printf("%s missing%c", obj_name, opt->output_delim); - break; - case SHORT_NAME_AMBIGUOUS: - printf("%s ambiguous%c", obj_name, opt->output_delim); - break; - case DANGLING_SYMLINK: - printf("dangling %"PRIuMAX"%c%s%c", - (uintmax_t)strlen(obj_name), - opt->output_delim, obj_name, opt->output_delim); - break; - case SYMLINK_LOOP: - printf("loop %"PRIuMAX"%c%s%c", - (uintmax_t)strlen(obj_name), - opt->output_delim, obj_name, opt->output_delim); - break; - case NOT_DIR: - printf("notdir %"PRIuMAX"%c%s%c", - (uintmax_t)strlen(obj_name), - opt->output_delim, obj_name, opt->output_delim); - break; - default: - BUG("unknown get_sha1_with_context result %d\n", - result); - break; + if (!opt->use_remote_info) { + result = get_oid_with_context(the_repository, obj_name, + flags, &data->oid, &ctx); + if (result != FOUND) { + switch (result) { + case MISSING_OBJECT: + printf("%s missing%c", obj_name, opt->output_delim); + break; + case SHORT_NAME_AMBIGUOUS: + printf("%s ambiguous%c", obj_name, opt->output_delim); + break; + case DANGLING_SYMLINK: + printf("dangling %"PRIuMAX"%c%s%c", + (uintmax_t)strlen(obj_name), + opt->output_delim, obj_name, opt->output_delim); + break; + case SYMLINK_LOOP: + printf("loop %"PRIuMAX"%c%s%c", + (uintmax_t)strlen(obj_name), + opt->output_delim, obj_name, opt->output_delim); + break; + case NOT_DIR: + printf("notdir %"PRIuMAX"%c%s%c", + (uintmax_t)strlen(obj_name), + opt->output_delim, obj_name, opt->output_delim); + break; + default: + BUG("unknown get_sha1_with_context result %d\n", + result); + break; + } + fflush(stdout); + return; } - fflush(stdout); - return; - } - if (ctx.mode == 0) { - printf("symlink %"PRIuMAX"%c%s%c", - (uintmax_t)ctx.symlink_path.len, - opt->output_delim, ctx.symlink_path.buf, opt->output_delim); - fflush(stdout); - return; + if (ctx.mode == 0) { + printf("symlink %"PRIuMAX"%c%s%c", + (uintmax_t)ctx.symlink_path.len, + opt->output_delim, ctx.symlink_path.buf, opt->output_delim); + fflush(stdout); + return; + } } batch_object_write(obj_name, scratch, opt, data, NULL, 0); } +static int get_remote_info(struct batch_options *opt, int argc, const char **argv) +{ + int retval = 0; + struct remote *remote = NULL; + struct object_id oid; + struct string_list object_info_options = STRING_LIST_INIT_NODUP; + static struct transport *gtransport; + + /* + * Change the format to "%(objectname) %(objectsize)" when + * remote-object-info command is used. Once we start supporting objecttype + * the default format should change to DEFAULT_FORMAT + */ + if (!opt->format) { + opt->format = "%(objectname) %(objectsize)"; + } + + remote = remote_get(argv[0]); + if (!remote) + die(_("must supply valid remote when using remote-object-info")); + oid_array_clear(&object_info_oids); + for (size_t i = 1; i < argc; i++) { + if (get_oid_hex(argv[i], &oid)) + die(_("Not a valid object name %s"), argv[i]); + oid_array_append(&object_info_oids, &oid); + } + + gtransport = transport_get(remote, NULL); + if (gtransport->smart_options) { + int include_size = 0; + + CALLOC_ARRAY(remote_object_info, object_info_oids.nr); + gtransport->smart_options->object_info = 1; + gtransport->smart_options->object_info_oids = &object_info_oids; + /* + * 'size' is the only option currently supported. + * Other options that are passed in the format will exit with error. + */ + if (strstr(opt->format, "%(objectsize)")) { + string_list_append(&object_info_options, "size"); + include_size = 1; + } + if (strstr(opt->format, "%(objecttype)")) { + die(_("objecttype is currently not supported with remote-object-info")); + } + if (strstr(opt->format, "%(objectsize:disk)")) + die(_("objectsize:disk is currently not supported with remote-object-info")); + if (strstr(opt->format, "%(deltabase)")) + die(_("deltabase is currently not supported with remote-object-info")); + if (object_info_options.nr > 0) { + gtransport->smart_options->object_info_options = &object_info_options; + for (size_t i = 0; i < object_info_oids.nr; i++) { + if (include_size) + remote_object_info[i].sizep = xcalloc(1, sizeof(long)); + } + gtransport->smart_options->object_info_data = &remote_object_info; + retval = transport_fetch_refs(gtransport, NULL); + } + } else { + retval = -1; + } + + return retval; +} + struct object_cb_data { struct batch_options *opt; struct expand_data *expand; @@ -642,6 +716,7 @@ typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *, struct queued_cmd { parse_cmd_fn_t fn; char *line; + const char *name; }; static void parse_cmd_contents(struct batch_options *opt, @@ -662,6 +737,55 @@ static void parse_cmd_info(struct batch_options *opt, batch_one_object(line, output, opt, data); } +static const struct parse_cmd { + const char *name; + parse_cmd_fn_t fn; + unsigned takes_args; +} commands[] = { + { "contents", parse_cmd_contents, 1 }, + { "info", parse_cmd_info, 1 }, + { "remote-object-info", parse_cmd_info, 1 }, + { "flush", NULL, 0 }, +}; + +static void parse_remote_info(struct batch_options *opt, + char *line, + struct strbuf *output, + struct expand_data *data, + const struct parse_cmd *p_cmd, + struct queued_cmd *q_cmd) +{ + int count; + const char **argv; + + count = split_cmdline(line, &argv); + if (get_remote_info(opt, count, argv)) + goto cleanup; + opt->use_remote_info = 1; + data->skip_object_info = 1; + data->mark_query = 0; + for (size_t i = 0; i < object_info_oids.nr; i++) { + if (remote_object_info[i].sizep) + data->size = *remote_object_info[i].sizep; + if (remote_object_info[i].typep) + data->type = *remote_object_info[i].typep; + + data->oid = object_info_oids.oid[i]; + if (p_cmd) + p_cmd->fn(opt, argv[i+1], output, data); + else + q_cmd->fn(opt, argv[i+1], output, data); + } + opt->use_remote_info = 0; + data->skip_object_info = 0; + data->mark_query = 1; + +cleanup: + for (size_t i = 0; i < object_info_oids.nr; i++) + free_object_info_contents(&remote_object_info[i]); + free(remote_object_info); +} + static void dispatch_calls(struct batch_options *opt, struct strbuf *output, struct expand_data *data, @@ -671,8 +795,12 @@ static void dispatch_calls(struct batch_options *opt, if (!opt->buffer_output) die(_("flush is only for --buffer mode")); - for (int i = 0; i < nr; i++) - cmd[i].fn(opt, cmd[i].line, output, data); + for (int i = 0; i < nr; i++) { + if (!strcmp(cmd[i].name, "remote-object-info")) + parse_remote_info(opt, cmd[i].line, output, data, NULL, &cmd[i]); + else + cmd[i].fn(opt, cmd[i].line, output, data); + } fflush(stdout); } @@ -685,17 +813,6 @@ static void free_cmds(struct queued_cmd *cmd, size_t *nr) *nr = 0; } - -static const struct parse_cmd { - const char *name; - parse_cmd_fn_t fn; - unsigned takes_args; -} commands[] = { - { "contents", parse_cmd_contents, 1}, - { "info", parse_cmd_info, 1}, - { "flush", NULL, 0}, -}; - static void batch_objects_command(struct batch_options *opt, struct strbuf *output, struct expand_data *data) @@ -740,11 +857,17 @@ static void batch_objects_command(struct batch_options *opt, dispatch_calls(opt, output, data, queued_cmd, nr); free_cmds(queued_cmd, &nr); } else if (!opt->buffer_output) { - cmd->fn(opt, p, output, data); + if (!strcmp(cmd->name, "remote-object-info")) { + char *line = xstrdup_or_null(p); + parse_remote_info(opt, line, output, data, cmd, NULL); + } else { + cmd->fn(opt, p, output, data); + } } else { ALLOC_GROW(queued_cmd, nr + 1, alloc); call.fn = cmd->fn; call.line = xstrdup_or_null(p); + call.name = cmd->name; queued_cmd[nr++] = call; } } @@ -761,8 +884,6 @@ static void batch_objects_command(struct batch_options *opt, strbuf_release(&input); } -#define DEFAULT_FORMAT "%(objectname) %(objecttype) %(objectsize)" - static int batch_objects(struct batch_options *opt) { struct strbuf input = STRBUF_INIT; diff --git a/object-file.c b/object-file.c index d3cf4b8b2e..6aaa167942 100644 --- a/object-file.c +++ b/object-file.c @@ -2988,3 +2988,14 @@ int read_loose_object(const char *path, munmap(map, mapsize); return ret; } + +void free_object_info_contents(struct object_info *object_info) +{ + if (!object_info) + return; + free(object_info->typep); + free(object_info->sizep); + free(object_info->disk_sizep); + free(object_info->delta_base_oid); + free(object_info->type_name); +} diff --git a/object-store-ll.h b/object-store-ll.h index c5f2bb2fc2..333e19cd1e 100644 --- a/object-store-ll.h +++ b/object-store-ll.h @@ -533,4 +533,7 @@ int for_each_object_in_pack(struct packed_git *p, int for_each_packed_object(each_packed_object_fn, void *, enum for_each_object_flags flags); +/* Free pointers inside of object_info, but not object_info itself */ +void free_object_info_contents(struct object_info *object_info); + #endif /* OBJECT_STORE_LL_H */ diff --git a/t/t1017-cat-file-remote-object-info.sh b/t/t1017-cat-file-remote-object-info.sh new file mode 100755 index 0000000000..7a7bdfeb91 --- /dev/null +++ b/t/t1017-cat-file-remote-object-info.sh @@ -0,0 +1,412 @@ +#!/bin/sh + +test_description='git cat-file --batch-command with remote-object-info command' + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +echo_without_newline () { + printf '%s' "$*" +} + +strlen () { + echo_without_newline "$1" | wc -c | sed -e 's/^ *//' +} + +hello_content="Hello World" +hello_size=$(strlen "$hello_content") +hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin) + +tree_size=$(($(test_oid rawsz) + 13)) + +commit_message="Initial commit" +commit_size=$(($(test_oid hexsz) + 137)) + +tag_header_without_oid="type blob +tag hellotag +tagger $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL>" +tag_header_without_timestamp="object $hello_oid +$tag_header_without_oid" +tag_description="This is a tag" +tag_content="$tag_header_without_timestamp 0 +0000 + +$tag_description" + +tag_oid=$(echo_without_newline "$tag_content" | git hash-object -t tag --stdin -w) +tag_size=$(strlen "$tag_content") + +# This section tests --batch-command with remote-object-info command +# Since "%(objecttype)" is currently not supported by the command remote-object-info , +# the filters are set to "%(objectname) %(objectsize)". +# Tests with the default filter are used to test the fallback to 'fetch' command + + +# Test --batch-command remote-object-info with 'git://' transport + +. "$TEST_DIRECTORY"/lib-git-daemon.sh +start_git_daemon --export-all --enable=receive-pack +daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent + +test_expect_success 'create repo to be served by git-daemon' ' + git init "$daemon_parent" && + + echo_without_newline "$hello_content" > $daemon_parent/hello && + git -C "$daemon_parent" update-index --add hello && + git -C "$daemon_parent" config transfer.advertiseobjectinfo true +' + +set_transport_variables () { + hello_sha1=$(echo_without_newline "$hello_content" | git hash-object --stdin) + tree_sha1=$(git -C "$1" write-tree) + commit_sha1=$(echo_without_newline "$commit_message" | git -C "$1" commit-tree $tree_sha1) + tag_sha1=$(echo_without_newline "$tag_content" | git -C "$1" hash-object -t tag --stdin -w) + tag_size=$(strlen "$tag_content") +} + + +test_expect_success 'batch-command remote-object-info git://' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent" && + + echo "$hello_sha1 $hello_size" >expect && + echo "$tree_sha1 $tree_size" >>expect && + echo "$commit_sha1 $commit_size" >>expect && + echo "$tag_sha1 $tag_size" >>expect && + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_sha1 + remote-object-info "$GIT_DAEMON_URL/parent" $tree_sha1 + remote-object-info "$GIT_DAEMON_URL/parent" $commit_sha1 + remote-object-info "$GIT_DAEMON_URL/parent" $tag_sha1 + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command remote-object-info git:// multiple sha1 per line' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent" && + + echo "$hello_sha1 $hello_size" >expect && + echo "$tree_sha1 $tree_size" >>expect && + echo "$commit_sha1 $commit_size" >>expect && + echo "$tag_sha1 $tag_size" >>expect && + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_sha1 $tree_sha1 $commit_sha1 $tag_sha1 + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command remote-object-info http:// default filter' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent" && + + echo "$hello_sha1 $hello_size" >expect && + echo "$tree_sha1 $tree_size" >>expect && + echo "$commit_sha1 $commit_size" >>expect && + echo "$tag_sha1 $tag_size" >>expect && + GIT_TRACE_PACKET=1 git cat-file --batch-command >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_sha1 $tree_sha1 + remote-object-info "$GIT_DAEMON_URL/parent" $commit_sha1 $tag_sha1 + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command --buffer remote-object-info git://' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent" && + + echo "$hello_sha1 $hello_size" >expect && + echo "$tree_sha1 $tree_size" >>expect && + echo "$commit_sha1 $commit_size" >>expect && + echo "$tag_sha1 $tag_size" >>expect && + git cat-file --batch-command="%(objectname) %(objectsize)" --buffer >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_sha1 $tree_sha1 + remote-object-info "$GIT_DAEMON_URL/parent" $commit_sha1 $tag_sha1 + flush + EOF + test_cmp expect actual + ) +' + +stop_git_daemon + +# Test --batch-command remote-object-info with 'http://' transport + +. "$TEST_DIRECTORY"/lib-httpd.sh +start_httpd + +test_expect_success 'create repo to be served by http:// transport' ' + git init "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" config http.receivepack true && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" config transfer.advertiseobjectinfo true && + echo_without_newline "$hello_content" > $HTTPD_DOCUMENT_ROOT_PATH/http_parent/hello && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" update-index --add hello +' + + +test_expect_success 'batch-command remote-object-info http://' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + + echo "$hello_sha1 $hello_size" >expect && + echo "$tree_sha1 $tree_size" >>expect && + echo "$commit_sha1 $commit_size" >>expect && + echo "$tag_sha1 $tag_size" >>expect && + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_sha1 + remote-object-info "$HTTPD_URL/smart/http_parent" $tree_sha1 + remote-object-info "$HTTPD_URL/smart/http_parent" $commit_sha1 + remote-object-info "$HTTPD_URL/smart/http_parent" $tag_sha1 + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command remote-object-info http:// one line' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + + echo "$hello_sha1 $hello_size" >expect && + echo "$tree_sha1 $tree_size" >>expect && + echo "$commit_sha1 $commit_size" >>expect && + echo "$tag_sha1 $tag_size" >>expect && + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_sha1 $tree_sha1 $commit_sha1 $tag_sha1 + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command --buffer remote-object-info http://' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + + echo "$hello_sha1 $hello_size" >expect && + echo "$tree_sha1 $tree_size" >>expect && + echo "$commit_sha1 $commit_size" >>expect && + echo "$tag_sha1 $tag_size" >>expect && + + git cat-file --batch-command="%(objectname) %(objectsize)" --buffer >actual <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_sha1 $tree_sha1 + remote-object-info "$HTTPD_URL/smart/http_parent" $commit_sha1 $tag_sha1 + flush + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command remote-object-info http:// default filter' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + + echo "$hello_sha1 $hello_size" >expect && + echo "$tree_sha1 $tree_size" >>expect && + echo "$commit_sha1 $commit_size" >>expect && + echo "$tag_sha1 $tag_size" >>expect && + + git cat-file --batch-command >actual <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_sha1 $tree_sha1 + remote-object-info "$HTTPD_URL/smart/http_parent" $commit_sha1 $tag_sha1 + EOF + test_cmp expect actual + ) +' + +test_expect_success 'remote-object-info fails on unspported filter option (objectsize:disk)' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + + test_must_fail git cat-file --batch-command="%(objectsize:disk)" 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_sha1 + EOF + test_grep "objectsize:disk is currently not supported with remote-object-info" err + ) +' + +test_expect_success 'remote-object-info fails on unspported filter option (deltabase)' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + + test_must_fail git cat-file --batch-command="%(deltabase)" 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_sha1 + EOF + test_grep "deltabase is currently not supported with remote-object-info" err + ) +' + +test_expect_success 'remote-object-info fails on server with legacy protocol' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + + test_must_fail git -c protocol.version=0 cat-file --batch-command="%(objectname) %(objectsize)" 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_sha1 + EOF + test_grep "remote-object-info requires protocol v2" err + ) +' + +test_expect_success 'remote-object-info fails on server with legacy protocol fallback' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + + test_must_fail git -c protocol.version=0 cat-file --batch-command 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_sha1 + EOF + test_grep "remote-object-info requires protocol v2" err + ) +' + +test_expect_success 'remote-object-info fails on malformed OID' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + malformed_object_id="this_id_is_not_valid" && + + test_must_fail git cat-file --batch-command="%(objectname) %(objectsize)" 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $malformed_object_id + EOF + test_grep "Not a valid object name '$malformed_object_id'" err + ) +' + +test_expect_success 'remote-object-info fails on malformed OID fallback' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + malformed_object_id="this_id_is_not_valid" && + + test_must_fail git cat-file --batch-command 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $malformed_object_id + EOF + test_grep "Not a valid object name '$malformed_object_id'" err + ) +' + +test_expect_success 'remote-object-info fails on missing OID' ' + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + git clone "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" missing_oid_repo && + test_commit -C missing_oid_repo message1 c.txt && + ( + cd missing_oid_repo && + + object_id=$(git rev-parse message1:c.txt) && + test_must_fail git cat-file --batch-command="%(objectname) %(objectsize)" 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $object_id + EOF + test_grep "object-info: not our ref $object_id" err + ) +' + +# shellcheck disable=SC2016 +test_expect_success 'remote-object-info fails on missing OID fallback' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd missing_oid_repo && + object_id=$(git rev-parse message1:c.txt) && + test_must_fail git cat-file --batch-command 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $object_id + EOF + test_grep "fatal: object-info: not our ref $object_id" err + ) +' + +# Test --batch-command remote-object-info with 'file://' transport + +# shellcheck disable=SC2016 +test_expect_success 'create repo to be served by file:// transport' ' + git init server && + git -C server config protocol.version 2 && + git -C server config transfer.advertiseobjectinfo true && + echo_without_newline "$hello_content" > server/hello && + git -C server update-index --add hello +' + + +test_expect_success 'batch-command remote-object-info file://' ' + ( + set_transport_variables "server" && + cd server && + + echo "$hello_sha1 $hello_size" >expect && + echo "$tree_sha1 $tree_size" >>expect && + echo "$commit_sha1 $commit_size" >>expect && + echo "$tag_sha1 $tag_size" >>expect && + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + remote-object-info "file://$(pwd)" $hello_sha1 + remote-object-info "file://$(pwd)" $tree_sha1 + remote-object-info "file://$(pwd)" $commit_sha1 + remote-object-info "file://$(pwd)" $tag_sha1 + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command remote-object-info file:// multiple sha1 per line' ' + ( + set_transport_variables "server" && + cd server && + + echo "$hello_sha1 $hello_size" >expect && + echo "$tree_sha1 $tree_size" >>expect && + echo "$commit_sha1 $commit_size" >>expect && + echo "$tag_sha1 $tag_size" >>expect && + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + remote-object-info "file://$(pwd)" $hello_sha1 $tree_sha1 $commit_sha1 $tag_sha1 + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command --buffer remote-object-info file://' ' + ( + set_transport_variables "server" && + cd server && + + echo "$hello_sha1 $hello_size" >expect && + echo "$tree_sha1 $tree_size" >>expect && + echo "$commit_sha1 $commit_size" >>expect && + echo "$tag_sha1 $tag_size" >>expect && + git cat-file --batch-command="%(objectname) %(objectsize)" --buffer >actual <<-EOF && + remote-object-info "file://$(pwd)" $hello_sha1 $tree_sha1 + remote-object-info "file://$(pwd)" $commit_sha1 $tag_sha1 + flush + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command remote-object-info file:// default filter' ' + ( + set_transport_variables "server" && + cd server && + + echo "$hello_sha1 $hello_size" >expect && + echo "$tree_sha1 $tree_size" >>expect && + echo "$commit_sha1 $commit_size" >>expect && + echo "$tag_sha1 $tag_size" >>expect && + + git cat-file --batch-command >actual <<-EOF && + remote-object-info "file://$(pwd)" $hello_sha1 $tree_sha1 + remote-object-info "file://$(pwd)" $commit_sha1 $tag_sha1 + EOF + test_cmp expect actual + ) +' + +test_done