From patchwork Tue Oct 6 22:05:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11819285 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF243618 for ; Tue, 6 Oct 2020 22:05:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B598D208C3 for ; Tue, 6 Oct 2020 22:05:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="vcYM/NFR" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727113AbgJFWFU (ORCPT ); Tue, 6 Oct 2020 18:05:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726002AbgJFWFS (ORCPT ); Tue, 6 Oct 2020 18:05:18 -0400 Received: from mail-wr1-x444.google.com (mail-wr1-x444.google.com [IPv6:2a00:1450:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 871EFC061755 for ; Tue, 6 Oct 2020 15:05:17 -0700 (PDT) Received: by mail-wr1-x444.google.com with SMTP id e17so6449582wru.12 for ; Tue, 06 Oct 2020 15:05:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Kt9nZJzCHemqRfq1pjAQFPykZqAgfCKSp+NhhOC/G5s=; b=vcYM/NFRv5L0tY4ICJ1TdJ4HkpW9rzLHl2YVueqq0u0ARHVGLlqiGyrBUmowerf2ec 8GJoDkiQ7A0jX6haXJ3Z2z9XFUS6jCM/B1W4TZV6JdG3aSQr2s0BKypw6iSCsXOWMEUY 0J2HEHdfh/mNidiJWcOBuJyYadRhJ81QZpBPVmxdJJGrq9/v6Iq4rEXKkSjsA/2NefQR /mCIrYMVV5bldgyfYyB9h4rL0Ss1xbixcsK6rk0gm07dkAL8tDTt83GYVio88+n/LMBe +gb3+4Wj3oDO2kVzgCTlJe5d7hw7czAd6a1l69ewA8EFNIOGeqs1hVEszFjER7BM2j4+ +VXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Kt9nZJzCHemqRfq1pjAQFPykZqAgfCKSp+NhhOC/G5s=; b=Yt9M5XMWFpMp2eXV+UZwdBiC1UB5vzVOHYAYxvBBcGzd8mDVKsQlhgDemkCZ56aCI0 DYOMYg8V2+a+oe+nrOcyW2t+ckg1dVHlhfLYuVYNhSKVg/Vv3fsqwSa74NQq2kRKvy2f YaJKa2/3ZpW3YzFIH9mbFxroRCUb0NexiecQVwYWYzBSHsO0FoorNz0FcrCMfJSWb8hE W5xSznbNgeTU3CnUDRLsPISkDm/nGfB65CxD/5xnbq42sV/XfEGdWDwI5pcKjc8ji55p Fz8rXeSCSxySa1AyAOVHZLvSif2fWeUc1A6iV0DSitRlXr7mXQtKwS6kpTrDMRLa/88k CerA== X-Gm-Message-State: AOAM532MjcW3XESO7PeIs6oZDTW6olw8mxgdhyjsPwTBpvFuS9ysAxog 0+jd6IQVC+A+zPsAglfplJG40Q2nLew= X-Google-Smtp-Source: ABdhPJxd2V0LNKGU45k2LEmv5XWyG158TX0JNM+/aHQpKXT5WZEH6LqMcL+1YPWH54PhCYL0+4IH+g== X-Received: by 2002:a5d:5387:: with SMTP id d7mr52991wrv.224.1602021916074; Tue, 06 Oct 2020 15:05:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n6sm168982wrx.58.2020.10.06.15.05.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Oct 2020 15:05:15 -0700 (PDT) Message-Id: <79b5f4a65197cea26ddc080c19dd2c5c7d424fc1.1602021913.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Tom Clarkson via GitGitGadget" Date: Tue, 06 Oct 2020 22:05:08 +0000 Subject: [PATCH v2 2/7] subtree: exclude commits predating add from recursive processing Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Avery Pennarun , Ed Maste , Johannes Schindelin , Tom Clarkson , Tom Clarkson Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Tom Clarkson Include recursion depth in debug logs so we can see when the recursion is getting out of hand. Making the cache handle null mappings correctly and adding older commits to the cache allows the recursive algorithm to terminate at any point on mainline rather than needing to reach either the add point or the initial commit. Signed-off-by: Tom Clarkson --- contrib/subtree/git-subtree.sh | 35 +++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh index 9867718503..160bad95c1 100755 --- a/contrib/subtree/git-subtree.sh +++ b/contrib/subtree/git-subtree.sh @@ -244,7 +244,7 @@ check_parents () { do if ! test -r "$cachedir/notree/$miss" then - debug " incorrect order: $miss" + debug " unprocessed parent commit: $miss ($indent)" process_split_commit "$miss" "" "$indent" fi done @@ -392,6 +392,24 @@ find_existing_splits () { done } +find_mainline_ref () { + debug "Looking for first split..." + dir="$1" + revs="$2" + + git log --reverse --grep="^git-subtree-dir: $dir/*\$" \ + --no-show-signature --pretty=format:'START %H%n%s%n%n%b%nEND%n' $revs | + while read a b junk + do + case "$a" in + git-subtree-mainline:) + echo "$b" + return + ;; + esac + done +} + copy_commit () { # We're going to set some environment vars here, so # do it in a subshell to get rid of them safely later @@ -646,9 +664,9 @@ process_split_commit () { progress "$revcount/$revmax ($createcount) [$extracount]" - debug "Processing commit: $rev" + debug "Processing commit: $rev ($indent)" exists=$(cache_get "$rev") - if test -n "$exists" + if test -z "$(cache_miss "$rev")" then debug " prior: $exists" return @@ -773,6 +791,17 @@ cmd_split () { unrevs="$(find_existing_splits "$dir" "$revs")" + mainline="$(find_mainline_ref "$dir" "$revs")" + if test -n "$mainline" + then + debug "Mainline $mainline predates subtree add" + git rev-list --topo-order --skip=1 $mainline | + while read rev + do + cache_set "$rev" "" + done || exit $? + fi + # We can't restrict rev-list to only $dir here, because some of our # parents have the $dir contents the root, and those won't match. # (and rev-list --follow doesn't seem to solve this) From patchwork Tue Oct 6 22:05:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11819283 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 62E35618 for ; Tue, 6 Oct 2020 22:05:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 36487208C3 for ; Tue, 6 Oct 2020 22:05:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nsSj4Zcf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727160AbgJFWFW (ORCPT ); Tue, 6 Oct 2020 18:05:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726681AbgJFWFS (ORCPT ); Tue, 6 Oct 2020 18:05:18 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69C41C0613D3 for ; Tue, 6 Oct 2020 15:05:18 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id z1so15132003wrt.3 for ; Tue, 06 Oct 2020 15:05:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7Re0tTXNL4gDIhFxT3jXKNLquiU/iqwZdKc/tizSqJc=; b=nsSj4ZcfPmwjI1gVZXB9AKG4mBslDLMy0blALE26ZacVpmhJkIwVNhRuhPlvzqJJ+g R06MDjrbcxP054YmRb669QxXzSPhzkgx2dsRA78Wz/ggkuFau0PaPU5kBTrZzOYQPRHI L+OwfMxOJXzZM0CZDLJAAefkLIudmuW+2W61rMMAR7rt/CbBC/yEz/Y++oJujSOVHbNq B607m791DopS7rOq4rqf6CWPkQM0NMpnmCSTSapOmycmo8RSMdP08hDT5o+oMNyr/xM7 Is/MONjVWZ7fRqbUhxDBtfY8FMAwIpeFNrWNGjmOcFdoFRJIjDzIy4ef5c5+PxTFYq99 vhLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7Re0tTXNL4gDIhFxT3jXKNLquiU/iqwZdKc/tizSqJc=; b=iD7bo3iKkN8Simuped4PAzYPfv3bWy3gNmHMkdEIwQXAGTh1DcwCEtV3qjB8RTHtb0 d5iiAxiRUxS+gXXXxdWVj8ZFmTziK61czlVwalPi67ixRZhwoaBgawyMQnDuK6dKzkUg WcUCi5wC+pXvaCRpicbJMZLrPI4lYlKupcroY12WV026dqpl6/ucjBwZJW20ZjozVuJo J6MsIMh8FoiywLKfuYhjBAfzUEjgBLJzkoU5vt87FbmXLQDAPD9Mhcx/jWnu7EaMqoCL keb3FLD++AY1ZV/gA/pFExOq0BEchUO+1sAeQgKmmYx1yeB7d18HJA8dIof7V28M0i0P 3j1g== X-Gm-Message-State: AOAM533caMuDrObBuyKpfHXK0uWzDo8tT6nk016SdgWw9EGfZu95kebT mDkLVNyDnc//b2N0ACXLC4JmZYOXLnY= X-Google-Smtp-Source: ABdhPJzfGk2JVK7Qe+seaiVSwtC5cJ9QiyGwBP6UcFWne2/KzSnI8ncxLXsSwZNDD9adke6505ukmA== X-Received: by 2002:adf:bbc6:: with SMTP id z6mr46759wrg.298.1602021916900; Tue, 06 Oct 2020 15:05:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o129sm107616wmb.25.2020.10.06.15.05.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Oct 2020 15:05:16 -0700 (PDT) Message-Id: <8eec18388c86071db47512b84118e3b9111bd34d.1602021913.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Tom Clarkson via GitGitGadget" Date: Tue, 06 Oct 2020 22:05:09 +0000 Subject: [PATCH v2 3/7] subtree: persist cache between split runs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Avery Pennarun , Ed Maste , Johannes Schindelin , Tom Clarkson , Tom Clarkson Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Tom Clarkson Provide a mechanism for handling problematic commits. If the algorithm in process_split_commit is getting something wrong, you can write a corrected value to the cache before running split. Signed-off-by: Tom Clarkson --- contrib/subtree/git-subtree.sh | 37 ++++++++++++++++++++++++++-------- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh index 160bad95c1..c21d620610 100755 --- a/contrib/subtree/git-subtree.sh +++ b/contrib/subtree/git-subtree.sh @@ -27,6 +27,7 @@ b,branch= create a new branch from the split subtree ignore-joins ignore prior --rejoin commits onto= try connecting new tree to an existing one rejoin merge the new branch back into HEAD +clear-cache reset the subtree mapping cache options for 'add', 'merge', and 'pull' squash merge subtree changes as a single commit " @@ -48,6 +49,7 @@ annotate= squash= message= prefix= +clearcache= debug () { if test -n "$debug" @@ -131,6 +133,9 @@ do --no-rejoin) rejoin= ;; + --clear-cache) + clearcache=1 + ;; --ignore-joins) ignore_joins=1 ;; @@ -206,9 +211,13 @@ debug "opts: {$*}" debug cache_setup () { - cachedir="$GIT_DIR/subtree-cache/$$" - rm -rf "$cachedir" || - die "Can't delete old cachedir: $cachedir" + cachedir="$GIT_DIR/subtree-cache/$prefix" + if test -n "$clearcache" + then + debug "Clearing cache" + rm -rf "$cachedir" || + die "Can't delete old cachedir: $cachedir" + fi mkdir -p "$cachedir" || die "Can't create new cachedir: $cachedir" mkdir -p "$cachedir/notree" || @@ -266,6 +275,16 @@ cache_set () { echo "$newrev" >"$cachedir/$oldrev" } +cache_set_if_unset () { + oldrev="$1" + newrev="$2" + if test -e "$cachedir/$oldrev" + then + return + fi + echo "$newrev" >"$cachedir/$oldrev" +} + rev_exists () { if git rev-parse "$1" >/dev/null 2>&1 then @@ -375,13 +394,13 @@ find_existing_splits () { then # squash commits refer to a subtree debug " Squash: $sq from $sub" - cache_set "$sq" "$sub" + cache_set_if_unset "$sq" "$sub" fi if test -n "$main" -a -n "$sub" then debug " Prior: $main -> $sub" - cache_set $main $sub - cache_set $sub $sub + cache_set_if_unset $main $sub + cache_set_if_unset $sub $sub try_remove_previous "$main" try_remove_previous "$sub" fi @@ -688,6 +707,8 @@ process_split_commit () { if test -n "$newparents" then cache_set "$rev" "$rev" + else + cache_set "$rev" "" fi return fi @@ -785,7 +806,7 @@ cmd_split () { # the 'onto' history is already just the subdir, so # any parent we find there can be used verbatim debug " cache: $rev" - cache_set "$rev" "$rev" + cache_set_if_unset "$rev" "$rev" done fi @@ -798,7 +819,7 @@ cmd_split () { git rev-list --topo-order --skip=1 $mainline | while read rev do - cache_set "$rev" "" + cache_set_if_unset "$rev" "" done || exit $? fi From patchwork Tue Oct 6 22:05:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11819273 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 826B3618 for ; Tue, 6 Oct 2020 22:05:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 55BA8208C7 for ; Tue, 6 Oct 2020 22:05:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VrURtn2h" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727119AbgJFWFU (ORCPT ); Tue, 6 Oct 2020 18:05:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726702AbgJFWFT (ORCPT ); Tue, 6 Oct 2020 18:05:19 -0400 Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 279FEC0613D4 for ; Tue, 6 Oct 2020 15:05:19 -0700 (PDT) Received: by mail-wm1-x342.google.com with SMTP id j136so353839wmj.2 for ; Tue, 06 Oct 2020 15:05:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=qHR3+3mHYmznNp1HGoR9hffJqdaCSV+4vNzb0cmV3ms=; b=VrURtn2h5D5CL1Y2aTaxFa7Ge3KTbaFLfT3THPQM+SuFLmQhJUWQgDJWlYKyKuZLYz oMNsCdJclqbAdzC6gf0oh4uVtuvJNEaNGP3VEcm8r+atUrRYEbYW+dLpoL41HFM2iy75 IIBUBlXm/WiFWzywS+RM7wuFRxOwEJSsng4a2EEmOakyFa8TgYiglTMhzP+An5/p+788 V5mHOOcaIcFwDpQm5K5M77S9sYSiscIIq1h1Q3PH8toqs3ju89IUpCwuOkVu2KxFwFuK JAeAhlH0Xkb4bvigoEzBjbmYMY2ZzsDtym1gDmTod5r8/KvJcewrrmYeYsdsbNsCmWKt 7dzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=qHR3+3mHYmznNp1HGoR9hffJqdaCSV+4vNzb0cmV3ms=; b=bdRrm7rgMN5HXNXDTmyMoxPldcA03qrkVsFdowUt19EvPxRWUFFqe61MPtiIuDw2t5 BK2uJQzi/J8BK48ncnOdx0V9x45vpXnMzry53vsLTxIxwURR0BZhvj4UrXaqfeOYszdn sbLTjc4oVWrexHaD1XeBqU503jls1XplCgAY7ky26S8bJHQKEgUEgQ5JD42tch2EGTlM kf30ABVWd5RMoHcT1gIHh9IPcgVt90lS4d9x+OwGuPqbPPkXiZa3T0vbqOe/Al6UOqCx F/Xtpdo8YAhNWIC2dAnmFVyajkgn5yQ69lc2Xxmkn/p0B9m1EeaLS6nRe3QpSlFa072G xfrg== X-Gm-Message-State: AOAM5308mkJ+o8hA6NMZbXMI2sqBudBd5/xtpLTfyVNAn/siA9au0RNs JjJo0xxftbSrxeCv1EAt3fU4QQ15xA0= X-Google-Smtp-Source: ABdhPJwVxnNktGj5P8TIwMvRwo3csr1Jsb7ZYZuPGV1tQRph8otL9l/ExIP1+xlST5wCOBWM95Kmcg== X-Received: by 2002:a1c:2b05:: with SMTP id r5mr11316wmr.179.1602021917723; Tue, 06 Oct 2020 15:05:17 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e7sm195686wrm.6.2020.10.06.15.05.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Oct 2020 15:05:17 -0700 (PDT) Message-Id: <1490ce111463b56eed35016c2e6232eea1faa3db.1602021913.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Tom Clarkson via GitGitGadget" Date: Tue, 06 Oct 2020 22:05:10 +0000 Subject: [PATCH v2 4/7] subtree: add git subtree map command Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Avery Pennarun , Ed Maste , Johannes Schindelin , Tom Clarkson , Tom Clarkson Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Tom Clarkson Adds an entry to the subtree cache so that subsequent split runs can skip any commits that turn out to be problematic. Signed-off-by: Tom Clarkson --- contrib/subtree/git-subtree.sh | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh index c21d620610..1559100c0e 100755 --- a/contrib/subtree/git-subtree.sh +++ b/contrib/subtree/git-subtree.sh @@ -15,6 +15,7 @@ git subtree merge --prefix= git subtree pull --prefix= git subtree push --prefix= git subtree split --prefix= +git subtree map --prefix= -- h,help show the help q quiet @@ -161,7 +162,7 @@ command="$1" shift case "$command" in -add|merge|pull) +add|merge|pull|map) default= ;; split|push) @@ -192,7 +193,8 @@ dir="$(dirname "$prefix/.")" if test "$command" != "pull" && test "$command" != "add" && - test "$command" != "push" + test "$command" != "push" && + test "$command" != "map" then revs=$(git rev-parse $default --revs-only "$@") || exit $? dirs=$(git rev-parse --no-revs --no-flags "$@") || exit $? @@ -793,6 +795,21 @@ cmd_add_commit () { say "Added dir '$dir'" } +cmd_map () { + oldrev="$1" + newrev="$2" + + if test -z "$oldrev" + then + die "You must provide a revision to map" + fi + + cache_setup || exit $? + cache_set "$oldrev" "$newrev" + + say "Mapped $oldrev => $newrev" +} + cmd_split () { debug "Splitting $dir..." cache_setup || exit $? From patchwork Tue Oct 6 22:05:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11819279 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6DF0C1580 for ; Tue, 6 Oct 2020 22:05:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 47B53208C3 for ; Tue, 6 Oct 2020 22:05:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bSOTVrNd" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727377AbgJFWFX (ORCPT ); Tue, 6 Oct 2020 18:05:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49584 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727100AbgJFWFU (ORCPT ); Tue, 6 Oct 2020 18:05:20 -0400 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E23A9C0613D5 for ; Tue, 6 Oct 2020 15:05:19 -0700 (PDT) Received: by mail-wr1-x442.google.com with SMTP id e17so6449658wru.12 for ; Tue, 06 Oct 2020 15:05:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pg4fNLNTjDSdd0vPCIQXtLYWhyk5ubFa4AGmgLOS+/o=; b=bSOTVrNdIIgtHQ8z7R4qXVJ4Yo8ES7XVpHEbqGv711QIL3SAYnv0GzNMAD3hIUW6x0 0oE0hgnb5BDKE7wZOcW1SRZb5d5XS7uXSq/P/ryDqVUbT3NVqyxcs2f1+Sad4DHj/0lJ MNhLVNiHjmqDqkXSJ5cYKn2dfvzUTe7Jj+ZbOtfLuqZ9JvKbGLCZUI1M6bMy1BcAC+zV jBZfrjRTJNtMLOzWOlR0lj1Fe+q1J17DY0/gaIVn7iMDN9MSbgN9HcPVLWhkGlJSL4fo 4s/KBeW3u+R8aDaTJ4WDjFPb1LCV/QxAv3Yw8M1CISvwE1BqgUCs1oEzx/qH682AR7BD C0jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pg4fNLNTjDSdd0vPCIQXtLYWhyk5ubFa4AGmgLOS+/o=; b=TNa298OFcrK0Hj9yBFTagQgNWlwefOnPMWBFnmQETD057gTE+kTpwnJp1nHBLLotBl 6uwxBDHhjizQzATB9jkYOzg5FTJK45Nh5ycg3hn7ShF57+shi2l3QeBAO0Ca/42+dyTb s93Jg/YU1e7/ZHa2q3bOK/0lwKxCjBKsQ9xh8NpfB0Rz05vt4cyd3x4MEs+oVweZ7PhN nUaElj8ZG6CHjPyu4/vn2jDU7rZisAfoprNNigWJEIYM1o2AqWUys94g+4rXYL0K0l5G jKwgk+x55UX4nv6/NWnoYNceur80VARao+H1buelM8v/VI8roKWkdhlwPtTG50ZcA+20 AYxg== X-Gm-Message-State: AOAM532yk2TLrL45Aru8VX4WNcOllXaqUi+8uaOITdZ7eZ+sfqSgIDNj zCAYRgQnfhh0npGzACjHCSOtTKPfYz8= X-Google-Smtp-Source: ABdhPJwAX1kO1U5qB6tR4Y3baTelFYoV5bKc3ZUl70JXEhIIJx2eJb9riUvEPtc/JuO/jx1oLmnFYQ== X-Received: by 2002:adf:ffc7:: with SMTP id x7mr11293wrs.405.1602021918483; Tue, 06 Oct 2020 15:05:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r14sm157709wrn.56.2020.10.06.15.05.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Oct 2020 15:05:18 -0700 (PDT) Message-Id: <2d103292cecfef3b29d221b3d8f05adb0add4475.1602021913.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Tom Clarkson via GitGitGadget" Date: Tue, 06 Oct 2020 22:05:11 +0000 Subject: [PATCH v2 5/7] subtree: add git subtree use and ignore commands Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Avery Pennarun , Ed Maste , Johannes Schindelin , Tom Clarkson , Tom Clarkson Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Tom Clarkson Tell split to use or ignore larger sections of the history. In most cases split does this automatically based on metadata from subtree add. Signed-off-by: Tom Clarkson --- contrib/subtree/git-subtree.sh | 78 ++++++++++++++++++++++++++++------ 1 file changed, 66 insertions(+), 12 deletions(-) diff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh index 1559100c0e..e56621a986 100755 --- a/contrib/subtree/git-subtree.sh +++ b/contrib/subtree/git-subtree.sh @@ -9,13 +9,15 @@ then set -- -h fi OPTS_SPEC="\ -git subtree add --prefix= -git subtree add --prefix= -git subtree merge --prefix= -git subtree pull --prefix= -git subtree push --prefix= -git subtree split --prefix= -git subtree map --prefix= +git subtree add --prefix= +git subtree add --prefix= +git subtree merge --prefix= +git subtree pull --prefix= +git subtree push --prefix= +git subtree split --prefix= +git subtree map --prefix= +git subtree ignore --prefix= +git subtree use --prefix= -- h,help show the help q quiet @@ -162,7 +164,7 @@ command="$1" shift case "$command" in -add|merge|pull|map) +add|merge|pull|map|ignore|use) default= ;; split|push) @@ -431,6 +433,18 @@ find_mainline_ref () { done } +exclude_processed_refs () { + if test -r "$cachedir/processed" + then + cat "$cachedir/processed" | + while read rev + do + debug "read $rev" + echo "^$rev" + done + fi +} + copy_commit () { # We're going to set some environment vars here, so # do it in a subshell to get rid of them safely later @@ -796,20 +810,60 @@ cmd_add_commit () { } cmd_map () { - oldrev="$1" - newrev="$2" - if test -z "$oldrev" + if test -z "$1" then die "You must provide a revision to map" fi + oldrev=$(git rev-parse --revs-only "$1") || exit $? + newrev= + + if test -n "$2" + then + newrev=$(git rev-parse --revs-only "$2") || exit $? + fi + cache_setup || exit $? cache_set "$oldrev" "$newrev" say "Mapped $oldrev => $newrev" } +cmd_ignore () { + revs=$(git rev-parse $default --revs-only "$@") || exit $? + ensure_single_rev $revs + + say "Ignoring $revs" + + cache_setup || exit $? + + git rev-list $revs | + while read rev + do + cache_set "$rev" "" + done + + echo "$revs" >>"$cachedir/processed" +} + +cmd_use () { + revs=$(git rev-parse $default --revs-only "$@") || exit $? + ensure_single_rev $revs + + say "Using existing subtree $revs" + + cache_setup || exit $? + + git rev-list $revs | + while read rev + do + cache_set "$rev" "$rev" + done + + echo "$revs" >>"$cachedir/processed" +} + cmd_split () { debug "Splitting $dir..." cache_setup || exit $? @@ -827,7 +881,7 @@ cmd_split () { done fi - unrevs="$(find_existing_splits "$dir" "$revs")" + unrevs="$(find_existing_splits "$dir" "$revs") $(exclude_processed_refs)" mainline="$(find_mainline_ref "$dir" "$revs")" if test -n "$mainline" From patchwork Tue Oct 6 22:05:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11819281 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B016E1920 for ; Tue, 6 Oct 2020 22:05:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 87C0F208C3 for ; Tue, 6 Oct 2020 22:05:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XXPuexN4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727449AbgJFWFX (ORCPT ); Tue, 6 Oct 2020 18:05:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727148AbgJFWFV (ORCPT ); Tue, 6 Oct 2020 18:05:21 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B43BBC061755 for ; Tue, 6 Oct 2020 15:05:20 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id l11so353884wmh.2 for ; Tue, 06 Oct 2020 15:05:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=02D01SreVWC3QQ327aPFNuK9f8RLXupMf7UWyHd6hwE=; b=XXPuexN4Km6v5tc3WRoi3oNcJ37J/x7PFRYq+k+CIVBietDkvoZVPQTkUgYzZo2oq3 cUPsb+bZJ363s3O/JpBw3vgOQN2Zv9q0MaUNxodm4Ls9nhhCK5dVoAC68fXaxDvcQ/S+ KMqVS/Df1gT+K0YykizkubenNZX6igB1atGF/sZos70oPmb7efN5kZde6Oun3Wv6smik qMMNhOwlp2VUR/n4QXwEC6vY1Qrk1xBVyol6/vg1y9ySSgUIZwdW6a1OQxkqFvu2Cxx1 drwhZxzQF+NoPM+v0toBSB5APipUzjMvTGYdpZrsPr1Pq4dFQqK25/wUrTHZz0D91IzT WcDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=02D01SreVWC3QQ327aPFNuK9f8RLXupMf7UWyHd6hwE=; b=OAhXpM/sjT7Ua0meWW7nFW3oOZXrGe+cBai/PNcZxIorXYfOTNOZ9bdZpAsdoxGS9i g4T2zK+N5cOVrftdiCFyBTLvXfQ7RbvWOPpCH7DziqPLFyGt0KnEt2gwerrQikymLEpT oQeWQUaXyGggscbfTkRaTcixgDFZoZBbzTcCcO5A9+ahIyBNpOV8DoXVQAz4Cy7n5uji yQfjzMOmDVOk6n1AXvT73ojOd4Fpc2mgHMMe6kuJ74oXskxo7iAJ9cpz2yTazm7h4Wwz aaoLCowYEUNqiSWt04ji/4m86m7GSHQ31wC/mk95MV1DWo2CNXVx2S2wpTlcMdJLejBj uQJg== X-Gm-Message-State: AOAM5332hCfm3OxXWz7Ch6+pY9ZqCgK9LuGrZq7YlZYTWzCsPXP8KPnL wKRyZqzZU2YWvtVXB05DDNYUXeE2T5A= X-Google-Smtp-Source: ABdhPJwyHapIcNB4z1Bfy85eVkteDdDaYQhX74HGSSuCKeqsauwKa2qmriKwFDvibj1uVmMUQRpIOQ== X-Received: by 2002:a1c:9cd3:: with SMTP id f202mr46880wme.148.1602021919290; Tue, 06 Oct 2020 15:05:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n65sm120029wmb.12.2020.10.06.15.05.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Oct 2020 15:05:18 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Tom Clarkson via GitGitGadget" Date: Tue, 06 Oct 2020 22:05:12 +0000 Subject: [PATCH v2 6/7] subtree: more robustly distinguish subtree and mainline commits Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Avery Pennarun , Ed Maste , Johannes Schindelin , Tom Clarkson , Tom Clarkson Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Tom Clarkson Prevent a mainline commit without $dir being treated as a subtree commit and pulling in the entire mainline history. Any valid subtree commit will have only valid subtree commits as parents, which will be unchanged by check_parents. Signed-off-by: Tom Clarkson --- contrib/subtree/git-subtree.sh | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh index e56621a986..fa6293b372 100755 --- a/contrib/subtree/git-subtree.sh +++ b/contrib/subtree/git-subtree.sh @@ -224,8 +224,6 @@ cache_setup () { fi mkdir -p "$cachedir" || die "Can't create new cachedir: $cachedir" - mkdir -p "$cachedir/notree" || - die "Can't create new cachedir: $cachedir/notree" debug "Using cachedir: $cachedir" >&2 } @@ -255,18 +253,11 @@ check_parents () { local indent=$(($2 + 1)) for miss in $missed do - if ! test -r "$cachedir/notree/$miss" - then - debug " unprocessed parent commit: $miss ($indent)" - process_split_commit "$miss" "" "$indent" - fi + debug " unprocessed parent commit: $miss ($indent)" + process_split_commit "$miss" "" "$indent" done } -set_notree () { - echo "1" > "$cachedir/notree/$1" -} - cache_set () { oldrev="$1" newrev="$2" @@ -719,11 +710,18 @@ process_split_commit () { # vs. a mainline commit? Does it matter? if test -z "$tree" then - set_notree "$rev" if test -n "$newparents" then - cache_set "$rev" "$rev" + if test "$newparents" = "$parents" + then + # if all parents were subtrees, this can be a subtree commit + cache_set "$rev" "$rev" + else + # a mainline commit with tree missing is equivalent to the initial commit + cache_set "$rev" "" + fi else + # no parents with valid subtree mappings means a commit prior to subtree add cache_set "$rev" "" fi return From patchwork Tue Oct 6 22:05:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11819287 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 463561580 for ; Tue, 6 Oct 2020 22:05:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1C3EE208C3 for ; Tue, 6 Oct 2020 22:05:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Pt+Mladt" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727518AbgJFWFc (ORCPT ); Tue, 6 Oct 2020 18:05:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727153AbgJFWFW (ORCPT ); Tue, 6 Oct 2020 18:05:22 -0400 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88AB7C061755 for ; Tue, 6 Oct 2020 15:05:21 -0700 (PDT) Received: by mail-wr1-x443.google.com with SMTP id e17so6449719wru.12 for ; Tue, 06 Oct 2020 15:05:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=BM5Qxo4wzcdQ2tkPzZekE+/HK7yjO7v3Zi8fVCPCIHE=; b=Pt+MladttjL7aVGw+HKJl4HFo4yh4EdJo9dlJY19nyZa5lO5/btkh3gGczXZEKTYLS M4dmaAaUf/jKvXKvjqLdbB/TBXSVsSQ2i5vdxdKpDTOF/PdfDH/0EpzUoGFpIrPuochl QrDLllxMQfwQ9um89GAaRFhtaMNQb8rUq/y4RY34z2GyYhU8LtTy20X6hiS8PZIvFlmQ lH2qy0xcvpC8v6+QlIAAVg0KvdKByVmH8OQGf8wWcdjNdgJNcjdPswwXyhcH35ahgTYW /+19ikmP8MLzuSJnL/Wc7x1HCB8ZT4pORVA5fWMWejAVRvdwmEtQe+FUHI47TsSgUTrU zPpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=BM5Qxo4wzcdQ2tkPzZekE+/HK7yjO7v3Zi8fVCPCIHE=; b=aQDd9xqfa5uQu3fSziqtY4CyuM4WadmbrCFRJtqvXvGlX5oOHm2ktxiVkdJLbhUdKR CGL28tv8deH+LLTxBbko3oW0LRQtgBVUAvt05aQ5CpXujogFvhmCn2kWmlL6yre6LCsU MdaKJgIlwfRy4jqH2xnkee7ZAGd/IU+Zvex+lSZlGQkluMJKplAjeQcPYjqnhSoXqve7 vT47sxdzWG9L94uXL8RC6bxqfkf3As9qs42iqDGFpT5Wt3g0L9Ko02xPgYSqm6C0xhwv TCftwNEHT1jsLqdhSlcOjYJ7gGpMsOqK1SwFzlENCtnG8g1lwgOcg512FaiJdWorfPIc PAhQ== X-Gm-Message-State: AOAM530hw//x5Iz4ZZCm/qIklSiHVCL2vJ0AOcBlGOEhdRihUj2koh8x DtSH8GdGlO3HSMnrQdbttDMRJ3JpJNc= X-Google-Smtp-Source: ABdhPJzcCTyEdCOSg1o39C7bCNfKaDQ/6PdnYVLe6lTxIR37oLn6K+Qy42PFiAabjlWZowxypywurw== X-Received: by 2002:adf:fbc5:: with SMTP id d5mr50610wrs.232.1602021920117; Tue, 06 Oct 2020 15:05:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c21sm87147wme.36.2020.10.06.15.05.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Oct 2020 15:05:19 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Tom Clarkson via GitGitGadget" Date: Tue, 06 Oct 2020 22:05:13 +0000 Subject: [PATCH v2 7/7] subtree: document new subtree commands Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Avery Pennarun , Ed Maste , Johannes Schindelin , Tom Clarkson , Tom Clarkson Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Tom Clarkson Signed-off-by: Tom Clarkson --- contrib/subtree/git-subtree.txt | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/contrib/subtree/git-subtree.txt b/contrib/subtree/git-subtree.txt index 352deda69d..a5a76e8ce6 100644 --- a/contrib/subtree/git-subtree.txt +++ b/contrib/subtree/git-subtree.txt @@ -52,6 +52,12 @@ useful elsewhere, you can extract its entire history and publish that as its own git repository, without accidentally intermingling the history of your application project. +Although the relationship between subtree and mainline commits is stored +in regular git history, it is also cached between subtree runs. In most +cases this is merely a performance improvement, but for projects with +large and complex histories the cache can be manipulated directly +with the use, ignore and map commands. + [TIP] In order to keep your commit messages clean, we recommend that people split their commits between the subtrees and the main @@ -120,6 +126,21 @@ and friends will work as expected. Note that if you use '--squash' when you merge, you should usually not just '--rejoin' when you split. +ignore:: + Mark a commit and all of its history as irrelevant to subtree split. + In most cases this would be handled automatically based on metadata + from subtree join commits. Intended for improving performance on + extremely large repos and excluding complex history that turns out + to be otherwise problematic. + +use:: + Mark a commit and all of its history as part of an existing subtree. + In normal circumstances this would be handled based on the metadata + from the subtree join commit. Similar to the --onto option of split. + +map:: + Manually override the normal output of split for a particular commit. + Extreme flexibility for advanced troubleshooting purposes only. OPTIONS ------- @@ -142,6 +163,9 @@ OPTIONS This option is only valid for add, merge and pull (unsure). Specify as the commit message for the merge commit. +--clear-cache:: + Reset the subtree cache and recalculate all subtree mappings from the + commit history OPTIONS FOR add, merge, push, pull ----------------------------------