From patchwork Wed Jul 12 12:12:24 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Vetter X-Patchwork-Id: 9836571 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id C1409603F3 for ; Wed, 12 Jul 2017 12:12:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B293C1FFCA for ; Wed, 12 Jul 2017 12:12:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A6F4828567; Wed, 12 Jul 2017 12:12:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED, T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6EEB61FFCA for ; Wed, 12 Jul 2017 12:12:38 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 007A86E408; Wed, 12 Jul 2017 12:12:35 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from mail-lf0-x244.google.com (mail-lf0-x244.google.com [IPv6:2a00:1450:4010:c07::244]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4D51D6E1DF for ; Wed, 12 Jul 2017 12:12:33 +0000 (UTC) Received: by mail-lf0-x244.google.com with SMTP id z78so2460200lff.2 for ; Wed, 12 Jul 2017 05:12:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aKXk3qlLyO7GZ3SEmp22jI+S9YOcAPlDYUJNmhUR43U=; b=KGW1BaOIdVEapx4XQR0PCuzYG6sq9KUBDR21P06lLyG+r3MzpWqoNkfoYisEHC8jWt witY960h+kiiWnj0Z6/OkpVwcgpDazraJRAQ+MQiX8k3P1hnsbS8/kXWQT0V+IRW69LQ OfonF24bG5nN9Fw55PlrUqQKqdL6fKUWsl3Rw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aKXk3qlLyO7GZ3SEmp22jI+S9YOcAPlDYUJNmhUR43U=; b=OYZ3wGk16CrI4Od6CD8Y8vUmXDgDpAEqfSPAga7Ldc0VkPMGMaHdVWrJZgiTMejuwU 6swZfbvQpWVNqRaiTj7q6NTwviSeImR2+7WvHp+bfTx/n3T29cBdFc7/4nEvm2DeuGZv GJddLlR/aBd543KAyAVDzATWXlMH2Q7f/hooZDnNXfOC7h43nIFLwO+IPeDPgsVUs2MM l6yxiNk71ypFGBmkCaXM53PskHa4TBg49rCFfMYMkzO0SxeID7ofXqqcqkK40u6PAeVk ehzddd1nQjPuN5qnYebwLfl2T08Rs3ZiDnuYkbx1+ighebIOaejgY4G/NqROySP2PpuD Fl4A== X-Gm-Message-State: AIVw112Qn5E4F/QKiJScnJnu99TpWlVfOXsMTwJe0RHydjKiadBMxRSM wm2y0d9idZXPfHkk X-Received: by 10.80.151.219 with SMTP id f27mr5839123edb.126.1499861551740; Wed, 12 Jul 2017 05:12:31 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:5640:0:960b:2678:e223:c1c6]) by smtp.gmail.com with ESMTPSA id t50sm1255365edb.41.2017.07.12.05.12.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Jul 2017 05:12:30 -0700 (PDT) From: Daniel Vetter To: DRI Development Date: Wed, 12 Jul 2017 14:12:24 +0200 Message-Id: <20170712121224.18522-2-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.13.2 In-Reply-To: <20170712121224.18522-1-daniel.vetter@ffwll.ch> References: <20170712121224.18522-1-daniel.vetter@ffwll.ch> Cc: Daniel Vetter , Intel Graphics Development , Daniel Vetter Subject: [Intel-gfx] [PATCH 2/2] dim: Try to gc the rr-cache X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Virus-Scanned: ClamAV using ClamSMTP The problem is that we have a distributed cache - every committer has a copy. Which means even just a slight clock skew will make sure that a naive gc algorithm results in lots of thrashing around. To fix this add a huge hysteresis: Only add files newer than 1 day, and only remove them when older than 60 days. As long as people have reasonable accurate clocks on their machines this should work. A different problem is that we can't use filesystem timestamps (and hence can't use git rerere gc): When someone comes back from vacations and updates git rerere, all the files will have current timestamps, even when they've been pushed out weeks ago. To fix that, use the git log to judge old files to remove. Also, remove old files before adding new ones, to avoid confusion. Also, we need to teach the cp -r to preserve timestamps, otherwise this won't work. v2: Use git log to remove old files. v3: Remove the debug uncommenting (Sean). v4: Split out code movement and explain better what's going on (Jani). Signed-off-by: Daniel Vetter Reviewed-by: Sean Paul --- dim | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/dim b/dim index b788edd29653..79d616cbf354 100755 --- a/dim +++ b/dim @@ -513,9 +513,15 @@ function commit_rerere_cache git pull >& /dev/null rm $(rr_cache_dir)/rr-cache -Rf &> /dev/null || true - cp $(rr_cache_dir)/* rr-cache -r + cp $(rr_cache_dir)/* rr-cache -r --preserve=timestamps git add ./*.patch >& /dev/null || true - git add rr-cache/* > /dev/null + for file in $(git ls-files); do + if ! git log --since="60 days ago" --name-only -- $file | grep $file &> /dev/null; then + git rm $file &> /dev/null + echo deleting $file + fi + done + find rr-cache/ -ctime -1 -type f -print0 | xargs -0 git add > /dev/null git rm rr-cache/rr-cache &> /dev/null || true if git commit -m "$time: $integration_branch rerere cache update" >& /dev/null; then echo -n "New commit. "