From patchwork Tue Jun 14 13:36:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Oliver X-Patchwork-Id: 12881171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DEE0C43334 for ; Tue, 14 Jun 2022 13:36:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243166AbiFNNgl (ORCPT ); Tue, 14 Jun 2022 09:36:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53178 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232585AbiFNNgj (ORCPT ); Tue, 14 Jun 2022 09:36:39 -0400 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2056.outbound.protection.outlook.com [40.107.220.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D8FB61C125 for ; Tue, 14 Jun 2022 06:36:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dIFCNd9RR5p+1F3aOge5T73ppVdcpbzV1TYyjYrxG9GG/o4n9uSJ/BQM4rgscU3yLwMBWrGTdWcUeZJi7cwq5E4pfZhCF3NlG9vSlvY71pRlKfSBNF69IEVh0DxC+0861YnwCyqd3uwo0/gqAWP3eq+4N4A2mCqgapLUL+EFeha/HulJKYEMnDdsNht/ZXhwAh9IMxUFUM+Os9n4sn+EqP98Zo43zMvLtX7kIhnl7+GdmYH6Omht/m9EtQzvXn/ruQ48BoXbDsxNh8CWsslKQI99aNKz2GXvSD2OHfEIbtXeXAEYUOK/eypbb7skvHL5mbv0ikuD1F2QXupfU8A1MA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lFARtGjiplzS3iULxvn90pdXIzRaJgmZKshtJUVdGCg=; b=bgU3deMUO96N66LFGzfwQ5Y1t7R/YODGxu46iHKZ1jtMNEkErreG7RS/FxobTpAzCgATR+TntwMi9KpzfvfxqaZAoCVwCZqaOGthlb8v+oBwtKFcHYJ9KzUg7hKKGHIWoER15CClZ/vSdDVwtUvpBU8qBhwDVkyxutGiBi/l2cgNhGaesxMCZPzMq9eRxciCtIw9QvEIX8z9rFGqwA6gcKzE40VTxgum/Pdio00yvlO4zZUBAcKJo0sEkAKNMSWHQp2BNOHlkUCxC9eHCiyJqhg82sg2VtCfNYlrGsGhwgPi4Xpljkkgv78z5vIUILATajfgDk1vMdYVO0OBOub38A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=roku.com; dmarc=pass action=none header.from=roku.com; dkim=pass header.d=roku.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=roku.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lFARtGjiplzS3iULxvn90pdXIzRaJgmZKshtJUVdGCg=; b=e8auKxTVdtyLdNUO6VBBAgHDtG6jVmpg9JdKPyGCxQMwBLmF8ycF6/i58xqDGU8A+FjO3O9mYtS4NRvXTyKGB89J2w8zdV/4UBDXnNZOpFTqHGRcmthhSI9FKI5pfZdVyYO1IYvCo5t4Inf8oKwDXpCsletOm6LdfJ4Th7NnaHvXn3LLCSrfGSZZzZz3JkWPp4oOYgrlbCswk+cqKqtsXxsTcxnRlL+rCBKEHFBm5GO7gYuZjRJ6OdUW8sLbk7SroyMhIw0u0Oatp/TTFYF3GfM6tMqZ1NUU1wxcj/klg3QDf+I32w8MrXF9aHQWatkxT0rpG6vTNoGey5qzSQdqxQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=roku.com; Received: from CO1PR01MB7225.prod.exchangelabs.com (2603:10b6:303:160::21) by BN0PR01MB6845.prod.exchangelabs.com (2603:10b6:408:14a::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.14; Tue, 14 Jun 2022 13:36:35 +0000 Received: from CO1PR01MB7225.prod.exchangelabs.com ([fe80::1062:158b:2993:5535]) by CO1PR01MB7225.prod.exchangelabs.com ([fe80::1062:158b:2993:5535%8]) with mapi id 15.20.5332.022; Tue, 14 Jun 2022 13:36:35 +0000 Message-ID: <77035a0f-c42e-5cb3-f422-03fe81093adb@roku.com> Date: Tue, 14 Jun 2022 14:36:28 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 To: git@vger.kernel.org Cc: jonathantanmy@google.com From: Richard Oliver Subject: [PATCH] mktree: learn about promised objects X-ClientProxiedBy: LO4P123CA0465.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1aa::20) To CO1PR01MB7225.prod.exchangelabs.com (2603:10b6:303:160::21) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 73621f28-4860-4b54-bb95-08da4e0ae65f X-MS-TrafficTypeDiagnostic: BN0PR01MB6845:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: lJ5/JAC0xbIrgcy6vPkyqFeT2+zw5XKNdzuTzfZgUhR/8GAJL/E1jVzBRe79wftM1CuKlydlcASVRPrvPDgv5G0aiuDwNQSdWxEBvUfmUkfDXeQU4IidTFEFhjdxawH/Ifc3izhRf+/mzq0LzevJyRTki5n2AtxLKLfMbvPoPB98vvszT6L4acwvZKVDkf1fRyGpvrShUH6iCvFKDCM5y4VCYA65OVmXRPvNkihOO81SU6utz6AN+Z7dLIC1Vhu6eA2wCqU719L0v671Vx33lnCs9/TwCyx9jBCyZi/7tSdWIg9PoXCKVbcsrh66k7FzLvEgogG93WhNCRFLQqSMNhrwMMYPkRsiKFzWYEoGmA+WjBiWUcx0BfJn/zNb4Kyd3MPaLgggqlRmu0gt1KjZOMWTDVGluLQS3nE8P2ZjzZfSThbr18KJbZnnjEan3UrpI7/6JttBCed+UIP8s35XjErs9xUzeuIBgzt2CDwiOmkTKp9gc2idEuaijC9OsAlHjAshNXtM4jmNJxf/ybwVBHMPSlMEL+FIwM4XaxDJxop8ySA5fuNs984t0tTkumyt8Ebj/2Lt0r/PrPLV+9rhxwPUs01zrn7SOSa+ZVJe1nieRLs1+LasTfnxYwpm9JU0KITeBojAY/PxygvY0pW+/yqSaGBbupHGtrOpCdgraBLtq41VDVvtSKAJZ7jPdC63jDmWFmRELmwnSpuy7AH43w== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO1PR01MB7225.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230016)(366004)(4326008)(6916009)(8936002)(2616005)(86362001)(316002)(8676002)(66476007)(31696002)(38100700002)(186003)(6666004)(66946007)(26005)(6506007)(6512007)(31686004)(66556008)(6486002)(83380400001)(508600001)(2906002)(36756003)(5660300002)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?DQXAX2bztAe29hiiEZWfb5DNZ3Pi?= =?utf-8?q?f4iJ6Xtqn9R/5SJK0B46g6m4oMbnJxHoyVc64czCDRX8DrpdLne80qgLut20F3SAI?= =?utf-8?q?nvVqExEgSO2q5qnQRsnz2JQJKzxBhuMosbPkRieRA6+cRqItAYOFMrP/w82UsuQfd?= =?utf-8?q?lZE3ZK087sWg6djCd6BMo/B3nYIB2lNEymCmBm3O2a6KoPnCcqQpi+g4cmtTu9Vo1?= =?utf-8?q?7cgP5ylXAVzbgMVfKMRmuE4JpDe22xL7Uvwb3a9sC9gdI6f6KFN3VLCEM1+d6UnsX?= =?utf-8?q?WJZ9PFMR4xXhQyMhoYjM3zQruxdJXTpEbY8nos5Be6VSRR15FYLTM9E8qemvFPgVP?= =?utf-8?q?aZ5tm/nF9zj8xknXFJy/UJ8hf2y8whK5jLmhHuQaeNDwTFxHvEZifEQ/a4PdqCq3w?= =?utf-8?q?bAhu5+TVLYzmHFJai8vHU40Xw4OFBV3/bxbiuSwXqlmG5voGLUYk94zhUOWlyPVRU?= =?utf-8?q?/gliCFD7zmDqqNO8aNWS4mhi0K0Yb36Lx6wOKUb2XvQKEoxLhKhawIlUcbY/crL6M?= =?utf-8?q?QiGGEEzEUDazRHAgQTVLUt4kKQwPrQc4d8j69nJCsJrjB4Q5g61rXCr6lI7dnXNmQ?= =?utf-8?q?bXfbcZfCSyc4/1jAvtdM08qgikUWxPhzm7f4At1RThxYOb4WK8oV+HtlRd53TSMT+?= =?utf-8?q?zcaOkIOmj/b6xq+JWrLm3hgp+Mh729XjWuwZ00W25g2ScRZb1+xsL4QmrKYIO/+VX?= =?utf-8?q?atxA6W2l3zP/D9FCZP69CZ4Zjyn6A65UfVJyZoEaaFaexfhZs2x5ISgOOGwz3jIwW?= =?utf-8?q?TXLo9L2BjgcpXATlWMlpcyB1MnW8feTPdc/rrVxXK4kX2y576hZteCPx27iG63Gzc?= =?utf-8?q?b9F1S/pUcoVsoG2782qazTVxixGbPUQYJT3uwnsQ/hI2jXc7b3zueRkGtmyBTmy7/?= =?utf-8?q?W//NZSDQYpXvUE3hLmkjUR82m9wVt4Qm0AoE0vVXkbIsPenKBLy1qyKRVVsOaLcgk?= =?utf-8?q?6UntNjbrl7B5QRCm5kopOWI06Q19EqZeuoSPbXMXQf7GnwsBiWhjBikAebXsOFu5o?= =?utf-8?q?CxB+yBjIkz6Im+oATKb/VG9MCayzWn+5DYQapJmaSs6gkPqdslmbCgV2b2TK94ClU?= =?utf-8?q?snoru/R3b8boxxadM0eTljPWdrOnXlz0zXD7YFVYjeh23mVRgSK9OdfZum2PEGygB?= =?utf-8?q?3Wcaut3fXgCe2SAgkNpSvmVMScPicue56/y1zrO7pzN+LnSkh+PoqsLmJP3ucVzFJ?= =?utf-8?q?K1rxvVe1Yi9eq8Jp5zg7yPkI1mFL/itxpVhkNx0OXJ6SckPWyndWR072JH01gaBkX?= =?utf-8?q?4GU4w2jUZsaub/3REFaAdQuMucEzPVDXhasfRsM7TXe+15J7C+zkTwDlK+d3WapFl?= =?utf-8?q?g0jW3AAfG8KqVti1vMGzuQdDf0wf29gAu7FomGpXyA4rW4t2JOaP0ZH91ffwrtuBY?= =?utf-8?q?UCNGpW09J5MoB6VxfXRQZKyb5vNF9DntkBkxh01dPe8LQQb9n2pG5c7iUWdi9IgHN?= =?utf-8?q?iqiL+MK7NsD5g0G85TOFXbPRKZv6GlxdzhhUs8NCGo9PK+H0h1uLAroHvpRy0051x?= =?utf-8?q?vpozm5k/zcZcaSeJeoS+o3HD7iQgL5R30qBOSozKtzN5FANkNqrIDq4D7hS0fG+Xp?= =?utf-8?q?+qQN2YfLXM6PGn3xyVvUfAJ5Lq4ve0NH6hIgVtXOlrqJXRCuG9FheRQ6Zu2JhikkM?= =?utf-8?q?7iD9xtSEDVovAgPAN6MNZjXakPfGch/w=3D=3D?= X-OriginatorOrg: roku.com X-MS-Exchange-CrossTenant-Network-Message-Id: 73621f28-4860-4b54-bb95-08da4e0ae65f X-MS-Exchange-CrossTenant-AuthSource: CO1PR01MB7225.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jun 2022 13:36:35.1768 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 00fc7e42-ad65-4c4c-ab54-848ba124a5b7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: iQ+or6jZbBUNPFfMTcEKvzghhVnYgtGSZYMDf9ume1HDossZewNGJ9L/xrnKNnzj42xrGSRl66kUokV4la5oqw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR01MB6845 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Do not use oid_object_info() to determine object type in mktree_line() as this can cause promised objects to be dynamically faulted-in one at a time which has poor performance. Instead, use a combination of oid_object_info_extended() (with OBJECT_INFO_SKIP_FETCH_OBJECT option), and the newly introduced promisor_object_type() to determine object type before defaulting to fetch from remote. Signed-off-by: Richard Oliver --- builtin/mktree.c | 19 ++++++++++++++-- packfile.c | 49 ++++++++++++++++++++++++++++------------ packfile.h | 6 +++++ t/t0410-partial-clone.sh | 28 +++++++++++++++++++++++ 4 files changed, 86 insertions(+), 16 deletions(-) base-commit: 1e59178e3f65880188caedb965e70db5ceeb2d64 diff --git a/builtin/mktree.c b/builtin/mktree.c index 902edba6d2..a783767b28 100644 --- a/builtin/mktree.c +++ b/builtin/mktree.c @@ -8,6 +8,7 @@ #include "tree.h" #include "parse-options.h" #include "object-store.h" +#include "packfile.h" static struct treeent { unsigned mode; @@ -116,8 +117,22 @@ static void mktree_line(char *buf, int nul_term_line, int allow_missing) path, ptr, type_name(mode_type)); } - /* Check the type of object identified by sha1 */ - obj_type = oid_object_info(the_repository, &oid, NULL); + /* Check the type of object identified by oid without fetching objects */ + struct object_info oi = OBJECT_INFO_INIT; + oi.typep = &obj_type; + if (oid_object_info_extended(the_repository, &oid, &oi, + OBJECT_INFO_LOOKUP_REPLACE | + OBJECT_INFO_SKIP_FETCH_OBJECT) < 0) + obj_type = -1; + + /* Has the object been promised to us if we can't find it */ + if (obj_type < 0) + obj_type = promisor_object_type(&oid); + + /* Try to fetch the object from the remote */ + if (obj_type < 0) + obj_type = oid_object_info(the_repository, &oid, NULL); + if (obj_type < 0) { if (allow_missing) { ; /* no problem - missing objects are presumed to be of the right type */ diff --git a/packfile.c b/packfile.c index 8e812a84a3..d27cb106c1 100644 --- a/packfile.c +++ b/packfile.c @@ -2223,17 +2223,26 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, return r ? r : pack_errors; } +static void map_put_type(kh_oid_pos_t *map, + const struct object_id *oid, + enum object_type type) +{ + int hash_ret; + khiter_t pos = kh_put_oid_pos(map, *oid, &hash_ret); + kh_value(map, pos) = type; +} + static int add_promisor_object(const struct object_id *oid, struct packed_git *pack, uint32_t pos, - void *set_) + void *map_) { - struct oidset *set = set_; + kh_oid_pos_t *map = map_; struct object *obj = parse_object(the_repository, oid); if (!obj) return 1; - oidset_insert(set, oid); + map_put_type(map, oid, obj->type); /* * If this is a tree, commit, or tag, the objects it refers @@ -2250,34 +2259,46 @@ static int add_promisor_object(const struct object_id *oid, */ return 0; while (tree_entry_gently(&desc, &entry)) - oidset_insert(set, &entry.oid); + map_put_type(map, + &entry.oid, + object_type(entry.mode)); free_tree_buffer(tree); } else if (obj->type == OBJ_COMMIT) { struct commit *commit = (struct commit *) obj; struct commit_list *parents = commit->parents; - oidset_insert(set, get_commit_tree_oid(commit)); + map_put_type(map, get_commit_tree_oid(commit), OBJ_TREE); for (; parents; parents = parents->next) - oidset_insert(set, &parents->item->object.oid); + map_put_type(map, + &parents->item->object.oid, + OBJ_COMMIT); } else if (obj->type == OBJ_TAG) { struct tag *tag = (struct tag *) obj; - oidset_insert(set, get_tagged_oid(tag)); + map_put_type(map, get_tagged_oid(tag), OBJ_COMMIT); } return 0; } -int is_promisor_object(const struct object_id *oid) +enum object_type promisor_object_type(const struct object_id *oid) { - static struct oidset promisor_objects; - static int promisor_objects_prepared; + static kh_oid_pos_t *promisor_objects; - if (!promisor_objects_prepared) { + if (!promisor_objects) { + promisor_objects = kh_init_oid_pos(); if (has_promisor_remote()) { for_each_packed_object(add_promisor_object, - &promisor_objects, + promisor_objects, FOR_EACH_OBJECT_PROMISOR_ONLY); } - promisor_objects_prepared = 1; } - return oidset_contains(&promisor_objects, oid); + khiter_t pos = kh_get_oid_pos(promisor_objects, *oid); + if (pos < kh_end(promisor_objects)) + return kh_value(promisor_objects, pos); + else + return OBJ_BAD; +} + +int is_promisor_object(const struct object_id *oid) +{ + return (promisor_object_type(oid) > OBJ_BAD); } diff --git a/packfile.h b/packfile.h index a3f6723857..35d0b600d2 100644 --- a/packfile.h +++ b/packfile.h @@ -182,6 +182,12 @@ int has_pack_index(const unsigned char *sha1); */ int is_promisor_object(const struct object_id *oid); +/* + * Return the object type if the given object is in, or referred to + * by, a promisor packfile; OBJ_BAD otherwise. + */ +enum object_type promisor_object_type(const struct object_id *oid); + /* * Expose a function for fuzz testing. * diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 1e864cf317..12bfb02187 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -660,6 +660,34 @@ test_expect_success 'lazy-fetch when accessing object not in the_repository' ' ! grep "[?]$FILE_HASH" out ' +test_expect_success 'missing blob object promised by tree, passes mktree' ' + rm -rf repo && + test_create_repo repo && + echo foo >repo/foo && + A=$(git -C repo hash-object foo) && + git -C repo update-index --add --info-only foo && + B=$(git -C repo write-tree --missing-ok) && + printf "$B\n" | pack_as_from_promisor && + delete_object repo "$B" && + + git -C repo config core.repositoryformatversion 1 && + git -C repo config extensions.partialclone "arbitrary string" && + + printf "100644 blob $A\tbar" | git -C repo mktree +' + +test_expect_success 'missing blob object, not promised, faulted-in by mktree' ' + test_create_repo server && + A=$(echo foo | git -C server hash-object --stdin -w) && + test_commit -C server 1 && + test_config -C server uploadpack.allowfilter 1 && + test_config -C server uploadpack.allowanysha1inwant 1 && + + rm -rf repo && + git clone --filter=blob:none "file://$(pwd)/server" repo && + printf "100644 blob $A\tbar" | git -C repo mktree +' + . "$TEST_DIRECTORY"/lib-httpd.sh start_httpd