From patchwork Wed Jul 12 12:12:24 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Vetter X-Patchwork-Id: 9836567 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id DA5AF602BD for ; Wed, 12 Jul 2017 12:12:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CBAF11FFCA for ; Wed, 12 Jul 2017 12:12:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C05C3285AF; Wed, 12 Jul 2017 12:12:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 737E21FFCA for ; Wed, 12 Jul 2017 12:12:37 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 95E4E6E404; Wed, 12 Jul 2017 12:12:34 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-lf0-x243.google.com (mail-lf0-x243.google.com [IPv6:2a00:1450:4010:c07::243]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8937A6E3FB for ; Wed, 12 Jul 2017 12:12:33 +0000 (UTC) Received: by mail-lf0-x243.google.com with SMTP id f28so2451915lfi.3 for ; Wed, 12 Jul 2017 05:12:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aKXk3qlLyO7GZ3SEmp22jI+S9YOcAPlDYUJNmhUR43U=; b=KGW1BaOIdVEapx4XQR0PCuzYG6sq9KUBDR21P06lLyG+r3MzpWqoNkfoYisEHC8jWt witY960h+kiiWnj0Z6/OkpVwcgpDazraJRAQ+MQiX8k3P1hnsbS8/kXWQT0V+IRW69LQ OfonF24bG5nN9Fw55PlrUqQKqdL6fKUWsl3Rw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aKXk3qlLyO7GZ3SEmp22jI+S9YOcAPlDYUJNmhUR43U=; b=T3WIMv+Y0KWOpdVPlSgzel++tZEBmvCMqk1rg/BNTr96Kl6F7INUTLOYKtOWNkx/LF MwgPP8/Dd9ElHcUu8RqUHnFn87UvdbWjoKPYTGYnUqfK9LF9yiq6na7mJEHt1S0VnqGz aogTIzrxN+CqGYMk/wnQpa5EJx8dUhWo9I/85KAyQ/U1SnXU8XlCwYtNTC5D1hoXUKKZ euiKMoLqNuD7bM1cpbhS+zxd8KRuIaCh0I+1n2Hq/hSb2CUswNNXHOk2NNbDMnTVjBeu +GXkvip0nHlnZoYthbBEDnuMBz3epci2i+vfqfui7PFTk/ed0kVJpEUtx4bXF+sQkQnZ rnXA== X-Gm-Message-State: AIVw111zdUSHwTAK5UR2CK4NPke1DgS4m2lXs4rD1WBkMB76zD5787ms I+1k1Lu8eaF1qnksGnI= X-Received: by 10.80.151.219 with SMTP id f27mr5839123edb.126.1499861551740; Wed, 12 Jul 2017 05:12:31 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:5640:0:960b:2678:e223:c1c6]) by smtp.gmail.com with ESMTPSA id t50sm1255365edb.41.2017.07.12.05.12.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Jul 2017 05:12:30 -0700 (PDT) From: Daniel Vetter To: DRI Development Subject: [PATCH 2/2] dim: Try to gc the rr-cache Date: Wed, 12 Jul 2017 14:12:24 +0200 Message-Id: <20170712121224.18522-2-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.13.2 In-Reply-To: <20170712121224.18522-1-daniel.vetter@ffwll.ch> References: <20170712121224.18522-1-daniel.vetter@ffwll.ch> Cc: Daniel Vetter , Intel Graphics Development , Daniel Vetter X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP The problem is that we have a distributed cache - every committer has a copy. Which means even just a slight clock skew will make sure that a naive gc algorithm results in lots of thrashing around. To fix this add a huge hysteresis: Only add files newer than 1 day, and only remove them when older than 60 days. As long as people have reasonable accurate clocks on their machines this should work. A different problem is that we can't use filesystem timestamps (and hence can't use git rerere gc): When someone comes back from vacations and updates git rerere, all the files will have current timestamps, even when they've been pushed out weeks ago. To fix that, use the git log to judge old files to remove. Also, remove old files before adding new ones, to avoid confusion. Also, we need to teach the cp -r to preserve timestamps, otherwise this won't work. v2: Use git log to remove old files. v3: Remove the debug uncommenting (Sean). v4: Split out code movement and explain better what's going on (Jani). Signed-off-by: Daniel Vetter Reviewed-by: Sean Paul --- dim | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/dim b/dim index b788edd29653..79d616cbf354 100755 --- a/dim +++ b/dim @@ -513,9 +513,15 @@ function commit_rerere_cache git pull >& /dev/null rm $(rr_cache_dir)/rr-cache -Rf &> /dev/null || true - cp $(rr_cache_dir)/* rr-cache -r + cp $(rr_cache_dir)/* rr-cache -r --preserve=timestamps git add ./*.patch >& /dev/null || true - git add rr-cache/* > /dev/null + for file in $(git ls-files); do + if ! git log --since="60 days ago" --name-only -- $file | grep $file &> /dev/null; then + git rm $file &> /dev/null + echo deleting $file + fi + done + find rr-cache/ -ctime -1 -type f -print0 | xargs -0 git add > /dev/null git rm rr-cache/rr-cache &> /dev/null || true if git commit -m "$time: $integration_branch rerere cache update" >& /dev/null; then echo -n "New commit. "