From patchwork Wed Jul 10 23:58:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11039097 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DDB7B13B1 for ; Wed, 10 Jul 2019 23:59:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD1FA289F9 for ; Wed, 10 Jul 2019 23:59:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C0AE828A00; Wed, 10 Jul 2019 23:59:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 36F02289F9 for ; Wed, 10 Jul 2019 23:59:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727622AbfGJX7g (ORCPT ); Wed, 10 Jul 2019 19:59:36 -0400 Received: from mail-qt1-f195.google.com ([209.85.160.195]:43556 "EHLO mail-qt1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727386AbfGJX7g (ORCPT ); Wed, 10 Jul 2019 19:59:36 -0400 Received: by mail-qt1-f195.google.com with SMTP id w17so4450818qto.10 for ; Wed, 10 Jul 2019 16:59:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CcQ7pP1ZfBMO1uGb5Oe17FrSCskIikm/t2kiS9FywJ8=; b=002Ma1gEwiwHoTRj98sw9ysPnGaVz4uOyVSZNuB2gG1G/mxhtdhVizJrL0AZBcKrcA gnGwUd7tV+2TC8koaPxaWDkSP4bSFH4osFkQWy7JgjWiaSsfFy0RQA3QrikqZSEt9iLW 81tLjL/mX+x2ZN2/CskdJC9JGRENpMqvi62mDgihigqx+syDgiNd+DHKTL9e6qc1Ib8V A0a26x5SLSEeC5q3KsJqrkwgbulwyITE9un/j6RUKkXvvvW85wJf38hKuE9B3RMfoP9t VeNrFjb1qLSLHPCDgzmreCDMZiFygjrhm+AeJnzUKzbrHWZGRhR0YuiaFjknbgOMibZg qXOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CcQ7pP1ZfBMO1uGb5Oe17FrSCskIikm/t2kiS9FywJ8=; b=rN/TMzovMYZDvBsFzstTikc+QGUNXqtQ39luKV1Y8C0wxOrOQ59rWOaWkbtuVuYZUi fFkJNvcq0QgPupU0Xm9OiXUF7vanmdTSCoGjxVICMpwcPQP7ZxO5+N3ft21X6WunVkMJ vketQ36qUtAGluvN0lAgcRiDpI05tL9et952rT0SqYM8zsXBoZp5nGd0AQi6qQHtf938 kmBX/G7JJzIadnoFOdQ4GZRo+/O/Z/1Ew93GH8GaXH2BeNjxrcFG4KEzZWmsMPVC2/Pq fBlXjsbuPxTcsTrnlxf5JXbkRkExi5/lokEmK35LOqqKYoVF2Te1uAxiRwr1FwxWFKzi 4DyQ== X-Gm-Message-State: APjAAAUQBoCwrjoeWmDwyPjrMSCTvK886pcfkAh7JoDn8Qc8pume0uDm jtJlKWqsgIksZQKFbOMop9flDQ== X-Google-Smtp-Source: APXvYqwjNdV8o25i/PLwuyYNQW9PD9Dl+8WujHbGzXqBCcihUeMvqqH4jMbf8MC9x7uDP7OHVJT3cw== X-Received: by 2002:a0c:aed0:: with SMTP id n16mr443881qvd.101.1562803173597; Wed, 10 Jul 2019 16:59:33 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id x8sm1714546qkl.27.2019.07.10.16.59.29 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 10 Jul 2019 16:59:33 -0700 (PDT) From: Matheus Tavares To: Junio C Hamano Cc: git@vger.kernel.org, Thomas Gummerer , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= , Christian Couder , =?utf-8?b?Tmd1eeG7hW4gVGg=?= =?utf-8?b?w6FpIE5n4buNYyBEdXk=?= , =?utf-8?q?SZEDER_G?= =?utf-8?q?=C3=A1bor?= , Olga Telezhnaya , Johannes Schindelin , kernel-usp@googlegroups.com, Alex Riesen Subject: [GSoC][PATCH v8 01/10] clone: test for our behavior on odd objects/* content Date: Wed, 10 Jul 2019 20:58:55 -0300 Message-Id: X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Ævar Arnfjörð Bjarmason Add tests for what happens when we perform a local clone on a repo containing odd files at .git/object directory, such as symlinks to other dirs, or unknown files. I'm bending over backwards here to avoid a SHA-1 dependency. See [1] for an earlier and simpler version that hardcoded SHA-1s. This behavior has been the same for a *long* time, but hasn't been tested for. There's a good post-hoc argument to be made for copying over unknown things, e.g. I'd like a git version that doesn't know about the commit-graph to copy it under "clone --local" so a newer git version can make use of it. In follow-up commits we'll look at changing some of this behavior, but for now, let's just assert it as-is so we'll notice what we'll change later. 1. https://public-inbox.org/git/20190226002625.13022-5-avarab@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason [matheus.bernardino: improved and split tests in more than one patch] Helped-by: Matheus Tavares Signed-off-by: Matheus Tavares --- t/t5604-clone-reference.sh | 111 +++++++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) diff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh index 4320082b1b..11250cab40 100755 --- a/t/t5604-clone-reference.sh +++ b/t/t5604-clone-reference.sh @@ -221,4 +221,115 @@ test_expect_success 'clone, dissociate from alternates' ' ( cd C && git fsck ) ' +test_expect_success 'setup repo with garbage in objects/*' ' + git init S && + ( + cd S && + test_commit A && + + cd .git/objects && + >.some-hidden-file && + >some-file && + mkdir .some-hidden-dir && + >.some-hidden-dir/some-file && + >.some-hidden-dir/.some-dot-file && + mkdir some-dir && + >some-dir/some-file && + >some-dir/.some-dot-file + ) +' + +test_expect_success 'clone a repo with garbage in objects/*' ' + for option in --local --no-hardlinks --shared --dissociate + do + git clone $option S S$option || return 1 && + git -C S$option fsck || return 1 + done && + find S-* -name "*some*" | sort >actual && + cat >expected <<-EOF && + S--dissociate/.git/objects/.some-hidden-file + S--dissociate/.git/objects/some-dir + S--dissociate/.git/objects/some-dir/.some-dot-file + S--dissociate/.git/objects/some-dir/some-file + S--dissociate/.git/objects/some-file + S--local/.git/objects/.some-hidden-file + S--local/.git/objects/some-dir + S--local/.git/objects/some-dir/.some-dot-file + S--local/.git/objects/some-dir/some-file + S--local/.git/objects/some-file + S--no-hardlinks/.git/objects/.some-hidden-file + S--no-hardlinks/.git/objects/some-dir + S--no-hardlinks/.git/objects/some-dir/.some-dot-file + S--no-hardlinks/.git/objects/some-dir/some-file + S--no-hardlinks/.git/objects/some-file + EOF + test_cmp expected actual +' + +test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' ' + git init T && + ( + cd T && + git config gc.auto 0 && + test_commit A && + git gc && + test_commit B && + + cd .git/objects && + mv pack packs && + ln -s packs pack && + find ?? -type d >loose-dirs && + last_loose=$(tail -n 1 loose-dirs) && + rm -f loose-dirs && + mv $last_loose a-loose-dir && + ln -s a-loose-dir $last_loose && + find . -type f | sort >../../../T.objects-files.raw && + echo unknown_content >unknown_file + ) && + git -C T fsck && + git -C T rev-list --all --objects >T.objects +' + + +test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' ' + for option in --local --no-hardlinks --shared --dissociate + do + git clone $option T T$option || return 1 && + git -C T$option fsck || return 1 && + git -C T$option rev-list --all --objects >T$option.objects && + test_cmp T.objects T$option.objects && + ( + cd T$option/.git/objects && + find . -type f | sort >../../../T$option.objects-files.raw + ) + done && + + for raw in $(ls T*.raw) + do + sed -e "s!/../!/Y/!; s![0-9a-f]\{38,\}!Z!" -e "/commit-graph/d" \ + -e "/multi-pack-index/d" <$raw >$raw.de-sha || return 1 + done && + + cat >expected-files <<-EOF && + ./Y/Z + ./Y/Z + ./a-loose-dir/Z + ./Y/Z + ./info/packs + ./pack/pack-Z.idx + ./pack/pack-Z.pack + ./packs/pack-Z.idx + ./packs/pack-Z.pack + ./unknown_file + EOF + + for option in --local --dissociate --no-hardlinks + do + test_cmp expected-files T$option.objects-files.raw.de-sha || return 1 + done && + + echo ./info/alternates >expected-files && + test_cmp expected-files T--shared.objects-files.raw +' + test_done From patchwork Wed Jul 10 23:58:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11039099 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 13C8E13B1 for ; Wed, 10 Jul 2019 23:59:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 04683289F9 for ; Wed, 10 Jul 2019 23:59:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ECB8C28A00; Wed, 10 Jul 2019 23:59:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 34DFF289F9 for ; Wed, 10 Jul 2019 23:59:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727681AbfGJX7s (ORCPT ); Wed, 10 Jul 2019 19:59:48 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:34260 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727386AbfGJX7s (ORCPT ); Wed, 10 Jul 2019 19:59:48 -0400 Received: by mail-qt1-f194.google.com with SMTP id k10so4507564qtq.1 for ; Wed, 10 Jul 2019 16:59:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=827o5jTxvqpUS+kthYBZwPX8/7/R8qFkd0KFNyuG3b8=; b=xjAVvVmZ1/alk4JgzpKrN9IEy8hK96HsaWF8byFczFEdFYopXSIE6GcF4hqPboilmc PXihJ0/ldEuXKSJpx5ZMZxOBgOiiMVwYQhn2n/il8Az7etJQwwROypafnjq2JKytr/MX BEOjmRSLFqEyN40eXrAfFnMUk+IgDe+l/GMKUl5dKt8/myhXENEBXxO8DelSK5fQWoc7 yjjZTbldJd8z2VyIjNBdLdZU80O5BuBoxtVQL/N3oue1RCIe7AE2q7RKXqsE3osuAlr7 vje7bbHgc0k3NtmVj8wRy63Q4A0ujPUPzELssQLIgyLqVHTl8xEkA4gmikAeZFbt43Zr 7Bag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=827o5jTxvqpUS+kthYBZwPX8/7/R8qFkd0KFNyuG3b8=; b=JLbzlY/Jk62x4yL4+vo1Wuu0mgPmMYInL5xKgFXN8HK65OC9cqtY6p14D9w2+IW5qR VPDV7oLGLhHn6JXOf2bdBU3hrcNcID/bW5SN8V131B7tmk/Yvck0/omIpeyx2RyRpLMf tvYzm7bKNiwWHXcf7RaoMNi2Zk/nGgHMgD7dadb14mwoLEA9rLzW8e7hL9pD8NWR4TqS w7btxLFqFJfqhkllz2z81z+n6cSD0HQkDd5XUrUgjDVAS1FZC0rG0dXoCS9lRS1KVwCf LzX4+sr+fBHlHcrt+n/qZM6eZeWcudtgzOProRhUufTRWBpCPEdMb3sALN7CKM0GG0zd eQpg== X-Gm-Message-State: APjAAAWN2XZNra+W1in4jGEo52uhMVHDx42kapeNJrDq3TPvQBIVcBAQ +Yyoj4UoPBOjSv4w2fi7UNQ9KA== X-Google-Smtp-Source: APXvYqzvjPfgNvlFT/HgNCBmHNXS7iuxQJhP6jlKtrrPnHY6SKZtj0CHxA/wcBW6Y9grlHdlAgYOKQ== X-Received: by 2002:ac8:30d2:: with SMTP id w18mr622082qta.296.1562803186668; Wed, 10 Jul 2019 16:59:46 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id x8sm1714546qkl.27.2019.07.10.16.59.40 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 10 Jul 2019 16:59:46 -0700 (PDT) From: Matheus Tavares To: Junio C Hamano Cc: git@vger.kernel.org, Thomas Gummerer , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= , Christian Couder , =?utf-8?b?Tmd1eeG7hW4gVGg=?= =?utf-8?b?w6FpIE5n4buNYyBEdXk=?= , =?utf-8?q?SZEDER_G?= =?utf-8?q?=C3=A1bor?= , Olga Telezhnaya , Johannes Schindelin , kernel-usp@googlegroups.com Subject: [GSoC][PATCH v8 02/10] clone: better handle symlinked files at .git/objects/ Date: Wed, 10 Jul 2019 20:58:56 -0300 Message-Id: <47a4f9b31c03499bc1317b9a0fccb11c2f5b4d34.1562801254.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP There is currently an odd behaviour when locally cloning a repository with symlinks at .git/objects: using --no-hardlinks all symlinks are dereferenced but without it, Git will try to hardlink the files with the link() function, which has an OS-specific behaviour on symlinks. On OSX and NetBSD, it creates a hardlink to the file pointed by the symlink whilst on GNU/Linux, it creates a hardlink to the symlink itself. On Manjaro GNU/Linux: $ touch a $ ln -s a b $ link b c $ ls -li a b c 155 [...] a 156 [...] b -> a 156 [...] c -> a But on NetBSD: $ ls -li a b c 2609160 [...] a 2609164 [...] b -> a 2609160 [...] c It's not good to have the result of a local clone to be OS-dependent and besides that, the current behaviour on GNU/Linux may result in broken symlinks. So let's standardize this by making the hardlinks always point to dereferenced paths, instead of the symlinks themselves. Also, add tests for symlinked files at .git/objects/. Note: Git won't create symlinks at .git/objects itself, but it's better to handle this case and be friendly with users who manually create them. Signed-off-by: Matheus Tavares Signed-off-by: Ævar Arnfjörð Bjarmason Co-authored-by: Ævar Arnfjörð Bjarmason --- builtin/clone.c | 2 +- t/t5604-clone-reference.sh | 27 ++++++++++++++++++++------- 2 files changed, 21 insertions(+), 8 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 5b9ebe9947..4a0a2455a7 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -445,7 +445,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest, if (unlink(dest->buf) && errno != ENOENT) die_errno(_("failed to unlink '%s'"), dest->buf); if (!option_no_hardlinks) { - if (!link(src->buf, dest->buf)) + if (!link(real_path(src->buf), dest->buf)) continue; if (option_local > 0) die_errno(_("failed to create link '%s'"), dest->buf); diff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh index 11250cab40..459ad8a20b 100755 --- a/t/t5604-clone-reference.sh +++ b/t/t5604-clone-reference.sh @@ -266,7 +266,7 @@ test_expect_success 'clone a repo with garbage in objects/*' ' test_cmp expected actual ' -test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknown files at objects/' ' +test_expect_success SYMLINKS 'setup repo with manually symlinked or unknown files at objects/' ' git init T && ( cd T && @@ -280,10 +280,19 @@ test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknow ln -s packs pack && find ?? -type d >loose-dirs && last_loose=$(tail -n 1 loose-dirs) && - rm -f loose-dirs && mv $last_loose a-loose-dir && ln -s a-loose-dir $last_loose && + first_loose=$(head -n 1 loose-dirs) && + rm -f loose-dirs && + + cd $first_loose && + obj=$(ls *) && + mv $obj ../an-object && + ln -s ../an-object $obj && + + cd ../ && find . -type f | sort >../../../T.objects-files.raw && + find . -type l | sort >../../../T.objects-symlinks.raw && echo unknown_content >unknown_file ) && git -C T fsck && @@ -291,7 +300,7 @@ test_expect_success SYMLINKS 'setup repo with manually symlinked dirs and unknow ' -test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files at objects/' ' +test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at objects/' ' for option in --local --no-hardlinks --shared --dissociate do git clone $option T T$option || return 1 && @@ -300,7 +309,8 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a test_cmp T.objects T$option.objects && ( cd T$option/.git/objects && - find . -type f | sort >../../../T$option.objects-files.raw + find . -type f | sort >../../../T$option.objects-files.raw && + find . -type l | sort >../../../T$option.objects-symlinks.raw ) done && @@ -314,6 +324,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a ./Y/Z ./Y/Z ./a-loose-dir/Z + ./an-object ./Y/Z ./info/packs ./pack/pack-Z.idx @@ -323,13 +334,15 @@ test_expect_success SYMLINKS 'clone repo with symlinked dirs and unknown files a ./unknown_file EOF - for option in --local --dissociate --no-hardlinks + for option in --local --no-hardlinks --dissociate do - test_cmp expected-files T$option.objects-files.raw.de-sha || return 1 + test_cmp expected-files T$option.objects-files.raw.de-sha || return 1 && + test_must_be_empty T$option.objects-symlinks.raw.de-sha || return 1 done && echo ./info/alternates >expected-files && - test_cmp expected-files T--shared.objects-files.raw + test_cmp expected-files T--shared.objects-files.raw && + test_must_be_empty T--shared.objects-symlinks.raw ' test_done From patchwork Wed Jul 10 23:58:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11039101 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E4E74138D for ; Wed, 10 Jul 2019 23:59:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D5E88289F9 for ; Wed, 10 Jul 2019 23:59:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C89D828A00; Wed, 10 Jul 2019 23:59:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4DD23289F9 for ; Wed, 10 Jul 2019 23:59:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727708AbfGJX7z (ORCPT ); Wed, 10 Jul 2019 19:59:55 -0400 Received: from mail-qk1-f193.google.com ([209.85.222.193]:35792 "EHLO mail-qk1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727386AbfGJX7z (ORCPT ); Wed, 10 Jul 2019 19:59:55 -0400 Received: by mail-qk1-f193.google.com with SMTP id r21so3421028qke.2 for ; Wed, 10 Jul 2019 16:59:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=80ph6NSFxNkHNnOpYOoHhUQRbAJvYajPVEg8eYowsDg=; b=QTHDIVSPPtA2EwE53DPF6I9fAuFR5RsX3YAMY2K99VOrkKfbUl7nCEUGk5iJaQ6Jyi Rz46Tmk/LR7xdbZoNp4A3cwtByiJjB6O/XoW0/leeA5ChL2gMg9rg6pSdqa4ubPhRlTT OWzy7f1nPNRbLPPG+Pdm3gAiqsPIK72HIPDhmI3fHJ8S2jPz/VnKIQtmAfNlIOuC6x0o 6ZZW/QdVDODUW6BnvgrdwRDNKEjM99BIAzEADIUqYEIMfRLvVi1hGJQBV0jUe8VL06x1 w7sOiKGrQ72sTYkieSvo5FHfEfSH/2TE+CA/vT6gA5n3qyLaikyxbVPufzvw1ECnIdVf OtVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=80ph6NSFxNkHNnOpYOoHhUQRbAJvYajPVEg8eYowsDg=; b=s9jh1WtN/mAKiAUvffF1mPR8r/BiqO1d4bvmcgtZa7S2G3nDUiGmk1XxGtI2ZFTBE/ Fe4Y1uC8Gm4e+MVQPRe+i+czqHvJSDr8QuDw8yAmpknJZSrYrcAmpxQTwW0zvbMPMC1s YwlQOEpwd/d4NbQQBXgW2XkSWbpsZ1K+bsAPv0MEZXR5dDkstRg/elZ07QjhBTMAMO51 8QfJKEC2Np3tE1O7fItrGeQsu9YXNi0v4spHtLJz+kdLH7WmmR6/kmGdiajFNoLySYrU eVDsdI4/qXHgJY4J5tjfKE02HF20ihIm9NmnD8cw0A/oSiPODP6l7SnpqV7WhevBX/53 O/Nw== X-Gm-Message-State: APjAAAUPx5Kn0m68obXAm2ZWoLAeNsrMkD/QGbs4WeSQexhgTX1ilBGH uL/YfQ057O31Gj2ydl2HIHzRSA== X-Google-Smtp-Source: APXvYqx2jvRFS2VLV/nZSuKRdLEJIgMPH9QEG0wraffPjcRWvQGl5e1h4hLQNVvdeyJyg9a5IgamnQ== X-Received: by 2002:a37:9944:: with SMTP id b65mr740950qke.105.1562803193819; Wed, 10 Jul 2019 16:59:53 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id x8sm1714546qkl.27.2019.07.10.16.59.48 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 10 Jul 2019 16:59:53 -0700 (PDT) From: Matheus Tavares To: Junio C Hamano Cc: git@vger.kernel.org, Thomas Gummerer , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= , Christian Couder , =?utf-8?b?Tmd1eeG7hW4gVGg=?= =?utf-8?b?w6FpIE5n4buNYyBEdXk=?= , =?utf-8?q?SZEDER_G?= =?utf-8?q?=C3=A1bor?= , Olga Telezhnaya , Johannes Schindelin , kernel-usp@googlegroups.com, Daniel Ferreira Subject: [GSoC][PATCH v8 03/10] dir-iterator: add tests for dir-iterator API Date: Wed, 10 Jul 2019 20:58:57 -0300 Message-Id: X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Daniel Ferreira Create t/helper/test-dir-iterator.c, which prints relevant information about a directory tree iterated over with dir-iterator. Create t/t0066-dir-iterator.sh, which tests that dir-iterator does iterate through a whole directory tree as expected. Signed-off-by: Daniel Ferreira [matheus.bernardino: update to use test-tool and some minor aesthetics] Helped-by: Matheus Tavares Signed-off-by: Matheus Tavares --- Makefile | 1 + t/helper/test-dir-iterator.c | 33 ++++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t0066-dir-iterator.sh | 55 ++++++++++++++++++++++++++++++++++++ 5 files changed, 91 insertions(+) create mode 100644 t/helper/test-dir-iterator.c create mode 100755 t/t0066-dir-iterator.sh diff --git a/Makefile b/Makefile index f58bf14c7b..7e2a44cccc 100644 --- a/Makefile +++ b/Makefile @@ -704,6 +704,7 @@ TEST_BUILTINS_OBJS += test-config.o TEST_BUILTINS_OBJS += test-ctype.o TEST_BUILTINS_OBJS += test-date.o TEST_BUILTINS_OBJS += test-delta.o +TEST_BUILTINS_OBJS += test-dir-iterator.o TEST_BUILTINS_OBJS += test-drop-caches.o TEST_BUILTINS_OBJS += test-dump-cache-tree.o TEST_BUILTINS_OBJS += test-dump-fsmonitor.o diff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c new file mode 100644 index 0000000000..84f50bed8c --- /dev/null +++ b/t/helper/test-dir-iterator.c @@ -0,0 +1,33 @@ +#include "test-tool.h" +#include "git-compat-util.h" +#include "strbuf.h" +#include "iterator.h" +#include "dir-iterator.h" + +/* Argument is a directory path to iterate over */ +int cmd__dir_iterator(int argc, const char **argv) +{ + struct strbuf path = STRBUF_INIT; + struct dir_iterator *diter; + + if (argc < 2) + die("BUG: test-dir-iterator needs one argument"); + + strbuf_add(&path, argv[1], strlen(argv[1])); + + diter = dir_iterator_begin(path.buf); + + while (dir_iterator_advance(diter) == ITER_OK) { + if (S_ISDIR(diter->st.st_mode)) + printf("[d] "); + else if (S_ISREG(diter->st.st_mode)) + printf("[f] "); + else + printf("[?] "); + + printf("(%s) [%s] %s\n", diter->relative_path, diter->basename, + diter->path.buf); + } + + return 0; +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 087a8c0cc9..7bc9bb231e 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -19,6 +19,7 @@ static struct test_cmd cmds[] = { { "ctype", cmd__ctype }, { "date", cmd__date }, { "delta", cmd__delta }, + { "dir-iterator", cmd__dir_iterator }, { "drop-caches", cmd__drop_caches }, { "dump-cache-tree", cmd__dump_cache_tree }, { "dump-fsmonitor", cmd__dump_fsmonitor }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index 7e703f3038..ec0ffbd0cb 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -9,6 +9,7 @@ int cmd__config(int argc, const char **argv); int cmd__ctype(int argc, const char **argv); int cmd__date(int argc, const char **argv); int cmd__delta(int argc, const char **argv); +int cmd__dir_iterator(int argc, const char **argv); int cmd__drop_caches(int argc, const char **argv); int cmd__dump_cache_tree(int argc, const char **argv); int cmd__dump_fsmonitor(int argc, const char **argv); diff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh new file mode 100755 index 0000000000..59bce868f4 --- /dev/null +++ b/t/t0066-dir-iterator.sh @@ -0,0 +1,55 @@ +#!/bin/sh + +test_description='Test the dir-iterator functionality' + +. ./test-lib.sh + +test_expect_success 'setup' ' + mkdir -p dir && + mkdir -p dir/a/b/c/ && + >dir/b && + >dir/c && + mkdir -p dir/d/e/d/ && + >dir/a/b/c/d && + >dir/a/e && + >dir/d/e/d/a && + + mkdir -p dir2/a/b/c/ && + >dir2/a/b/c/d +' + +test_expect_success 'dir-iterator should iterate through all files' ' + cat >expected-iteration-sorted-output <<-EOF && + [d] (a) [a] ./dir/a + [d] (a/b) [b] ./dir/a/b + [d] (a/b/c) [c] ./dir/a/b/c + [d] (d) [d] ./dir/d + [d] (d/e) [e] ./dir/d/e + [d] (d/e/d) [d] ./dir/d/e/d + [f] (a/b/c/d) [d] ./dir/a/b/c/d + [f] (a/e) [e] ./dir/a/e + [f] (b) [b] ./dir/b + [f] (c) [c] ./dir/c + [f] (d/e/d/a) [a] ./dir/d/e/d/a + EOF + + test-tool dir-iterator ./dir >out && + sort out >./actual-iteration-sorted-output && + + test_cmp expected-iteration-sorted-output actual-iteration-sorted-output +' + +test_expect_success 'dir-iterator should list files in the correct order' ' + cat >expected-pre-order-output <<-EOF && + [d] (a) [a] ./dir2/a + [d] (a/b) [b] ./dir2/a/b + [d] (a/b/c) [c] ./dir2/a/b/c + [f] (a/b/c/d) [d] ./dir2/a/b/c/d + EOF + + test-tool dir-iterator ./dir2 >actual-pre-order-output && + + test_cmp expected-pre-order-output actual-pre-order-output +' + +test_done From patchwork Wed Jul 10 23:58:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11039103 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D54FC13B1 for ; Thu, 11 Jul 2019 00:00:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C548828A00 for ; Thu, 11 Jul 2019 00:00:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B91A728A09; Thu, 11 Jul 2019 00:00:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5819328A00 for ; Thu, 11 Jul 2019 00:00:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727770AbfGKAAC (ORCPT ); Wed, 10 Jul 2019 20:00:02 -0400 Received: from mail-qt1-f193.google.com ([209.85.160.193]:41583 "EHLO mail-qt1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727386AbfGKAAC (ORCPT ); Wed, 10 Jul 2019 20:00:02 -0400 Received: by mail-qt1-f193.google.com with SMTP id d17so4444366qtj.8 for ; Wed, 10 Jul 2019 17:00:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CqQHOlwlfENkwc05/cG9Y2S77go+PAKki+Y1xBT2URM=; b=RRWyFdCd2a82Hp4pLzXPS0wiDaKzGQOzgx16pqvmGkF9djkP50Ugxp8K5DnczETPsv hF4wfFUfYas7/MIK9ByZwdKoAJ+4zPwJMMTNa9Z1zbmxnEy721iCO06vjfM1nTvPt+g4 tbaA3iGeQUUDlrUgAH8wWK1tkeA+DV6aiJWtfIOOmmQH8mwrultXF99ca0KzNkaPBPrD Y4BdhKr2r0vQT9FuGjCpIY1zAGYQt2SiUE8BbCy6M45ekrMYzYtw2GY0l+mAbTtwIs1e QPh2D/VIGyL+L3gozJwDW4Z4XtdcYWGOQwTOKWMtOKjXV/wbWFdlL1NFcrra0PVexkPp XXtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CqQHOlwlfENkwc05/cG9Y2S77go+PAKki+Y1xBT2URM=; b=KZpiVoFDGhnt9LEJViNQHhNTbwIjQTAJBIovPDTRnLfYX8rWUkWH3Pi3kYCtCB+SZ6 KQ6p5ASDEMkkJOZdZASG2VAp6QVj3Cnxz+lfsuoD5oxsMs1Ro7hLpVxspWCsPoj4Mb8Z mmgtiZWuRhQAxuQAt1+7/ljwCBkDDCq8WyYpUa0EY0eNNjMTfSx/Wr3gJSi/h4yZSg5p +HB4uCcauIxx6EBZTQa8/2yxEz66/ZNuLjaHUFOnGW9LPr8p5joa9kdr/5AciC9PB/6A xTBQfWu4TcNZ+5mSPIOiMlY1bAI7cqrAl845j9qP4GWcbaFWu1RivkVPxwAsQRSLZ8+Q XkKg== X-Gm-Message-State: APjAAAW2U1GLUxrmKJezDHvZAp6I6Aa6ckyun0POmlzeP4i4tHbd+r3J 89NHKK6dgPO8DVXFza1OB8YiTw== X-Google-Smtp-Source: APXvYqxNzC9TMZUDGLLFLK5RgAWz8v4oao5YRBO5+nKKJ10bhV3Z5Ko9S3lU47JuaZK7PMuq6iVetQ== X-Received: by 2002:ac8:2f43:: with SMTP id k3mr605431qta.179.1562803201116; Wed, 10 Jul 2019 17:00:01 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id x8sm1714546qkl.27.2019.07.10.16.59.55 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 10 Jul 2019 17:00:00 -0700 (PDT) From: Matheus Tavares To: Junio C Hamano Cc: git@vger.kernel.org, Thomas Gummerer , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= , Christian Couder , =?utf-8?b?Tmd1eeG7hW4gVGg=?= =?utf-8?b?w6FpIE5n4buNYyBEdXk=?= , =?utf-8?q?SZEDER_G?= =?utf-8?q?=C3=A1bor?= , Olga Telezhnaya , Johannes Schindelin , kernel-usp@googlegroups.com, Michael Haggerty Subject: [GSoC][PATCH v8 04/10] dir-iterator: use warning_errno when possible Date: Wed, 10 Jul 2019 20:58:58 -0300 Message-Id: <0cc5f1f0b4ea7de4e0508316e861ace50f39de1f.1562801255.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Change warning(..., strerror(errno)) by warning_errno(...). This helps to unify warning display besides simplifying a bit the code. Also, improve warning messages by surrounding paths with quotation marks and using more meaningful statements. Signed-off-by: Matheus Tavares --- dir-iterator.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/dir-iterator.c b/dir-iterator.c index f2dcd82fde..0c8880868a 100644 --- a/dir-iterator.c +++ b/dir-iterator.c @@ -71,8 +71,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator) level->dir = opendir(iter->base.path.buf); if (!level->dir && errno != ENOENT) { - warning("error opening directory %s: %s", - iter->base.path.buf, strerror(errno)); + warning_errno("error opening directory '%s'", + iter->base.path.buf); /* Popping the level is handled below */ } @@ -122,11 +122,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator) if (!de) { /* This level is exhausted; pop up a level. */ if (errno) { - warning("error reading directory %s: %s", - iter->base.path.buf, strerror(errno)); + warning_errno("error reading directory '%s'", + iter->base.path.buf); } else if (closedir(level->dir)) - warning("error closing directory %s: %s", - iter->base.path.buf, strerror(errno)); + warning_errno("error closing directory '%s'", + iter->base.path.buf); level->dir = NULL; if (--iter->levels_nr == 0) @@ -140,9 +140,8 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator) strbuf_addstr(&iter->base.path, de->d_name); if (lstat(iter->base.path.buf, &iter->base.st) < 0) { if (errno != ENOENT) - warning("error reading path '%s': %s", - iter->base.path.buf, - strerror(errno)); + warning_errno("failed to stat '%s'", + iter->base.path.buf); continue; } @@ -170,9 +169,11 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator) &iter->levels[iter->levels_nr - 1]; if (level->dir && closedir(level->dir)) { + int saved_errno = errno; strbuf_setlen(&iter->base.path, level->prefix_len); - warning("error closing directory %s: %s", - iter->base.path.buf, strerror(errno)); + errno = saved_errno; + warning_errno("error closing directory '%s'", + iter->base.path.buf); } } From patchwork Wed Jul 10 23:58:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11039105 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5D9086C5 for ; Thu, 11 Jul 2019 00:00:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D28828A00 for ; Thu, 11 Jul 2019 00:00:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 412EE28A09; Thu, 11 Jul 2019 00:00:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2E4BF28A00 for ; Thu, 11 Jul 2019 00:00:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727774AbfGKAAK (ORCPT ); Wed, 10 Jul 2019 20:00:10 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:44266 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727386AbfGKAAK (ORCPT ); Wed, 10 Jul 2019 20:00:10 -0400 Received: by mail-qt1-f194.google.com with SMTP id 44so4436266qtg.11 for ; Wed, 10 Jul 2019 17:00:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8OtkvmcFp/O262g880vfrtg9QEzDp0VMVxCp+lomV4Y=; b=TmqMdKzdDIvxuHDbB2J/ay1oQb9TTIq06YZ5nXWclwcuIgBtTeeTSIncC/zVi79mTM wfsI50M5pbQdnX2cPkBjIpaRoaJ2PsjRwmNoCUGBfnfSQgef4gylQAUS458Bhfy/6lTg KRLWLJUN3eiDCw07o2KGqoKuknKriLcwBm62opcbihpkd4W0bgV/SAf4Itoz5Fpxlrzk iB1SK5nBaSWxSUP1nl4IM2XuqldcR4XFWE897iJLeOWZ5KOIkPOvrl6YSB7YrD489jom IwMwK0VrE8/bbxNa3QR9BfOx7nn27gZcYurZ/5fQu7TLUqy+G6Of/sd33IbdpVZNRG6g /SAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8OtkvmcFp/O262g880vfrtg9QEzDp0VMVxCp+lomV4Y=; b=W6hIL/xhqbXNs/3NodHSsbL3W7dIj8yWYR6Svug+S4xD8b8luOvHeABJ5PoKfTNs6n Ryp4RukDPgX77Eo6Q9HOXO6Ay+CuNy2QWKkFje1/uYyEp/oQ3N1zF8JU/8Xa1y18UtWv obTrN5Cp9zFVtSSlA0LwPr0V95f0B97PxFacl28PfRsN1Tf1qhX6X/bbJGhVo9yHpoSX jUg6PRm7HdIhDRanQGR5iEhWUiPztbGvyi8w+22JroFgEh1HwPhFWqwc8UTD2HnhMMIE ln5tqFvOTEQNJuekrdEq/3yrKR/7J1yGjVzZAj1FKR2nLZpj3mv/gCXjP32uRuWBBh+9 9Q6g== X-Gm-Message-State: APjAAAXADcgsD7Q6GwqhJDWLuiRQRFmz+ZbAARmeNujzdDJfVwmdYekC hDxIscONT0dBM7arbgZZtW9CCg== X-Google-Smtp-Source: APXvYqzA2mdm4QL7R5jDoL8LDuVzqFdxpxn/Py8J9WwtjphNllNULortpb77licn2xmEWV8Q7gdojA== X-Received: by 2002:ac8:7cd:: with SMTP id m13mr594851qth.341.1562803208279; Wed, 10 Jul 2019 17:00:08 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id x8sm1714546qkl.27.2019.07.10.17.00.02 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 10 Jul 2019 17:00:07 -0700 (PDT) From: Matheus Tavares To: Junio C Hamano Cc: git@vger.kernel.org, Thomas Gummerer , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= , Christian Couder , =?utf-8?b?Tmd1eeG7hW4gVGg=?= =?utf-8?b?w6FpIE5n4buNYyBEdXk=?= , =?utf-8?q?SZEDER_G?= =?utf-8?q?=C3=A1bor?= , Olga Telezhnaya , Johannes Schindelin , kernel-usp@googlegroups.com, Daniel Ferreira , Jeff King , Ramsay Jones , Michael Haggerty Subject: [GSoC][PATCH v8 05/10] dir-iterator: refactor state machine model Date: Wed, 10 Jul 2019 20:58:59 -0300 Message-Id: X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP dir_iterator_advance() is a large function with two nested loops. Let's improve its readability factoring out three functions and simplifying its mechanics. The refactored model will no longer depend on level.initialized and level.dir_state to keep track of the iteration state and will perform on a single loop. Also, dir_iterator_begin() currently does not check if the given string represents a valid directory path. Since the refactored model will have to stat() the given path at initialization, let's also check for this kind of error and make dir_iterator_begin() return NULL, on failures, with errno appropriately set. And add tests for this new behavior. Improve documentation at dir-iteration.h and code comments at dir-iterator.c to reflect the changes and eliminate possible ambiguities. Finally, adjust refs/files-backend.c to check for now possible dir_iterator_begin() failures. Original-patch-by: Daniel Ferreira Signed-off-by: Matheus Tavares --- dir-iterator.c | 234 ++++++++++++++++++----------------- dir-iterator.h | 17 ++- refs/files-backend.c | 17 ++- t/helper/test-dir-iterator.c | 5 + t/t0066-dir-iterator.sh | 13 ++ 5 files changed, 164 insertions(+), 122 deletions(-) diff --git a/dir-iterator.c b/dir-iterator.c index 0c8880868a..594fe4d67b 100644 --- a/dir-iterator.c +++ b/dir-iterator.c @@ -4,8 +4,6 @@ #include "dir-iterator.h" struct dir_iterator_level { - int initialized; - DIR *dir; /* @@ -13,16 +11,6 @@ struct dir_iterator_level { * (including a trailing '/'): */ size_t prefix_len; - - /* - * The last action that has been taken with the current entry - * (needed for directories, which have to be included in the - * iteration and also iterated into): - */ - enum { - DIR_STATE_ITER, - DIR_STATE_RECURSE - } dir_state; }; /* @@ -34,9 +22,11 @@ struct dir_iterator_int { struct dir_iterator base; /* - * The number of levels currently on the stack. This is always - * at least 1, because when it becomes zero the iteration is - * ended and this struct is freed. + * The number of levels currently on the stack. After the first + * call to dir_iterator_begin(), if it succeeds to open the + * first level's dir, this will always be at least 1. Then, + * when it comes to zero the iteration is ended and this + * struct is freed. */ size_t levels_nr; @@ -50,113 +40,118 @@ struct dir_iterator_int { struct dir_iterator_level *levels; }; +/* + * Push a level in the iter stack and initialize it with information from + * the directory pointed by iter->base->path. It is assumed that this + * strbuf points to a valid directory path. Return 0 on success and -1 + * otherwise, leaving the stack unchanged. + */ +static int push_level(struct dir_iterator_int *iter) +{ + struct dir_iterator_level *level; + + ALLOC_GROW(iter->levels, iter->levels_nr + 1, iter->levels_alloc); + level = &iter->levels[iter->levels_nr++]; + + if (!is_dir_sep(iter->base.path.buf[iter->base.path.len - 1])) + strbuf_addch(&iter->base.path, '/'); + level->prefix_len = iter->base.path.len; + + level->dir = opendir(iter->base.path.buf); + if (!level->dir) { + if (errno != ENOENT) { + warning_errno("error opening directory '%s'", + iter->base.path.buf); + } + iter->levels_nr--; + return -1; + } + + return 0; +} + +/* + * Pop the top level on the iter stack, releasing any resources associated + * with it. Return the new value of iter->levels_nr. + */ +static int pop_level(struct dir_iterator_int *iter) +{ + struct dir_iterator_level *level = + &iter->levels[iter->levels_nr - 1]; + + if (level->dir && closedir(level->dir)) + warning_errno("error closing directory '%s'", + iter->base.path.buf); + level->dir = NULL; + + return --iter->levels_nr; +} + +/* + * Populate iter->base with the necessary information on the next iteration + * entry, represented by the given dirent de. Return 0 on success and -1 + * otherwise. + */ +static int prepare_next_entry_data(struct dir_iterator_int *iter, + struct dirent *de) +{ + strbuf_addstr(&iter->base.path, de->d_name); + /* + * We have to reset these because the path strbuf might have + * been realloc()ed at the previous strbuf_addstr(). + */ + iter->base.relative_path = iter->base.path.buf + + iter->levels[0].prefix_len; + iter->base.basename = iter->base.path.buf + + iter->levels[iter->levels_nr - 1].prefix_len; + + if (lstat(iter->base.path.buf, &iter->base.st)) { + if (errno != ENOENT) + warning_errno("failed to stat '%s'", iter->base.path.buf); + return -1; + } + + return 0; +} + int dir_iterator_advance(struct dir_iterator *dir_iterator) { struct dir_iterator_int *iter = (struct dir_iterator_int *)dir_iterator; + if (S_ISDIR(iter->base.st.st_mode)) { + if (push_level(iter) && iter->levels_nr == 0) { + /* Pushing the first level failed */ + return dir_iterator_abort(dir_iterator); + } + } + + /* Loop until we find an entry that we can give back to the caller. */ while (1) { + struct dirent *de; struct dir_iterator_level *level = &iter->levels[iter->levels_nr - 1]; - struct dirent *de; - if (!level->initialized) { - /* - * Note: dir_iterator_begin() ensures that - * path is not the empty string. - */ - if (!is_dir_sep(iter->base.path.buf[iter->base.path.len - 1])) - strbuf_addch(&iter->base.path, '/'); - level->prefix_len = iter->base.path.len; - - level->dir = opendir(iter->base.path.buf); - if (!level->dir && errno != ENOENT) { - warning_errno("error opening directory '%s'", + strbuf_setlen(&iter->base.path, level->prefix_len); + errno = 0; + de = readdir(level->dir); + + if (!de) { + if (errno) + warning_errno("error reading directory '%s'", iter->base.path.buf); - /* Popping the level is handled below */ - } - - level->initialized = 1; - } else if (S_ISDIR(iter->base.st.st_mode)) { - if (level->dir_state == DIR_STATE_ITER) { - /* - * The directory was just iterated - * over; now prepare to iterate into - * it. - */ - level->dir_state = DIR_STATE_RECURSE; - ALLOC_GROW(iter->levels, iter->levels_nr + 1, - iter->levels_alloc); - level = &iter->levels[iter->levels_nr++]; - level->initialized = 0; - continue; - } else { - /* - * The directory has already been - * iterated over and iterated into; - * we're done with it. - */ - } + else if (pop_level(iter) == 0) + return dir_iterator_abort(dir_iterator); + continue; } - if (!level->dir) { - /* - * This level is exhausted (or wasn't opened - * successfully); pop up a level. - */ - if (--iter->levels_nr == 0) - return dir_iterator_abort(dir_iterator); + if (is_dot_or_dotdot(de->d_name)) + continue; + if (prepare_next_entry_data(iter, de)) continue; - } - /* - * Loop until we find an entry that we can give back - * to the caller: - */ - while (1) { - strbuf_setlen(&iter->base.path, level->prefix_len); - errno = 0; - de = readdir(level->dir); - - if (!de) { - /* This level is exhausted; pop up a level. */ - if (errno) { - warning_errno("error reading directory '%s'", - iter->base.path.buf); - } else if (closedir(level->dir)) - warning_errno("error closing directory '%s'", - iter->base.path.buf); - - level->dir = NULL; - if (--iter->levels_nr == 0) - return dir_iterator_abort(dir_iterator); - break; - } - - if (is_dot_or_dotdot(de->d_name)) - continue; - - strbuf_addstr(&iter->base.path, de->d_name); - if (lstat(iter->base.path.buf, &iter->base.st) < 0) { - if (errno != ENOENT) - warning_errno("failed to stat '%s'", - iter->base.path.buf); - continue; - } - - /* - * We have to set these each time because - * the path strbuf might have been realloc()ed. - */ - iter->base.relative_path = - iter->base.path.buf + iter->levels[0].prefix_len; - iter->base.basename = - iter->base.path.buf + level->prefix_len; - level->dir_state = DIR_STATE_ITER; - - return ITER_OK; - } + return ITER_OK; } } @@ -187,17 +182,32 @@ struct dir_iterator *dir_iterator_begin(const char *path) { struct dir_iterator_int *iter = xcalloc(1, sizeof(*iter)); struct dir_iterator *dir_iterator = &iter->base; - - if (!path || !*path) - BUG("empty path passed to dir_iterator_begin()"); + int saved_errno; strbuf_init(&iter->base.path, PATH_MAX); strbuf_addstr(&iter->base.path, path); ALLOC_GROW(iter->levels, 10, iter->levels_alloc); + iter->levels_nr = 0; - iter->levels_nr = 1; - iter->levels[0].initialized = 0; + /* + * Note: stat already checks for NULL or empty strings and + * inexistent paths. + */ + if (stat(iter->base.path.buf, &iter->base.st) < 0) { + saved_errno = errno; + goto error_out; + } + + if (!S_ISDIR(iter->base.st.st_mode)) { + saved_errno = ENOTDIR; + goto error_out; + } return dir_iterator; + +error_out: + dir_iterator_abort(dir_iterator); + errno = saved_errno; + return NULL; } diff --git a/dir-iterator.h b/dir-iterator.h index 970793d07a..9b4cb7acd2 100644 --- a/dir-iterator.h +++ b/dir-iterator.h @@ -8,18 +8,22 @@ * * Iterate over a directory tree, recursively, including paths of all * types and hidden paths. Skip "." and ".." entries and don't follow - * symlinks except for the original path. + * symlinks except for the original path. Note that the original path + * is not included in the iteration. * * Every time dir_iterator_advance() is called, update the members of * the dir_iterator structure to reflect the next path in the * iteration. The order that paths are iterated over within a - * directory is undefined, but directory paths are always iterated - * over before the subdirectory contents. + * directory is undefined, directory paths are always given before + * their contents. * * A typical iteration looks like this: * * int ok; - * struct iterator *iter = dir_iterator_begin(path); + * struct dir_iterator *iter = dir_iterator_begin(path); + * + * if (!iter) + * goto error_handler; * * while ((ok = dir_iterator_advance(iter)) == ITER_OK) { * if (want_to_stop_iteration()) { @@ -59,8 +63,9 @@ struct dir_iterator { }; /* - * Start a directory iteration over path. Return a dir_iterator that - * holds the internal state of the iteration. + * Start a directory iteration over path. On success, return a + * dir_iterator that holds the internal state of the iteration. + * In case of failure, return NULL and set errno accordingly. * * The iteration includes all paths under path, not including path * itself and not including "." or ".." entries. diff --git a/refs/files-backend.c b/refs/files-backend.c index 63e55e6773..7ed81046d4 100644 --- a/refs/files-backend.c +++ b/refs/files-backend.c @@ -2143,13 +2143,22 @@ static struct ref_iterator_vtable files_reflog_iterator_vtable = { static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store, const char *gitdir) { - struct files_reflog_iterator *iter = xcalloc(1, sizeof(*iter)); - struct ref_iterator *ref_iterator = &iter->base; + struct dir_iterator *diter; + struct files_reflog_iterator *iter; + struct ref_iterator *ref_iterator; struct strbuf sb = STRBUF_INIT; - base_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0); strbuf_addf(&sb, "%s/logs", gitdir); - iter->dir_iterator = dir_iterator_begin(sb.buf); + + diter = dir_iterator_begin(sb.buf); + if(!diter) + return empty_ref_iterator_begin(); + + iter = xcalloc(1, sizeof(*iter)); + ref_iterator = &iter->base; + + base_ref_iterator_init(ref_iterator, &files_reflog_iterator_vtable, 0); + iter->dir_iterator = diter; iter->ref_store = ref_store; strbuf_release(&sb); diff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c index 84f50bed8c..fab1ff6237 100644 --- a/t/helper/test-dir-iterator.c +++ b/t/helper/test-dir-iterator.c @@ -17,6 +17,11 @@ int cmd__dir_iterator(int argc, const char **argv) diter = dir_iterator_begin(path.buf); + if (!diter) { + printf("dir_iterator_begin failure: %d\n", errno); + exit(EXIT_FAILURE); + } + while (dir_iterator_advance(diter) == ITER_OK) { if (S_ISDIR(diter->st.st_mode)) printf("[d] "); diff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh index 59bce868f4..cc4b19c34c 100755 --- a/t/t0066-dir-iterator.sh +++ b/t/t0066-dir-iterator.sh @@ -52,4 +52,17 @@ test_expect_success 'dir-iterator should list files in the correct order' ' test_cmp expected-pre-order-output actual-pre-order-output ' +test_expect_success 'begin should fail upon inexistent paths' ' + test_must_fail test-tool dir-iterator ./inexistent-path \ + >actual-inexistent-path-output && + echo "dir_iterator_begin failure: 2" >expected-inexistent-path-output && + test_cmp expected-inexistent-path-output actual-inexistent-path-output +' + +test_expect_success 'begin should fail upon non directory paths' ' + test_must_fail test-tool dir-iterator ./dir/b >actual-non-dir-output && + echo "dir_iterator_begin failure: 20" >expected-non-dir-output && + test_cmp expected-non-dir-output actual-non-dir-output +' + test_done From patchwork Wed Jul 10 23:59:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11039107 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 097986C5 for ; Thu, 11 Jul 2019 00:00:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EE05228A00 for ; Thu, 11 Jul 2019 00:00:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E260028A09; Thu, 11 Jul 2019 00:00:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DFAB828A00 for ; Thu, 11 Jul 2019 00:00:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727783AbfGKAAT (ORCPT ); Wed, 10 Jul 2019 20:00:19 -0400 Received: from mail-qk1-f195.google.com ([209.85.222.195]:36407 "EHLO mail-qk1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727386AbfGKAAS (ORCPT ); Wed, 10 Jul 2019 20:00:18 -0400 Received: by mail-qk1-f195.google.com with SMTP id g18so3410173qkl.3 for ; Wed, 10 Jul 2019 17:00:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QnmmwIJNP9kkVUAS3cBc0nlBHoiLXYNHoaymUDNzq/A=; b=nMKZeqYKILfgDHITJOw9BelDx4gBYSuykPnnjiEmpxOBn/xHb5Xu2Ua2OjnncJTMIy nF+TTBaHt47gPnkPqvR9zHbtXqlwLPgTNH7rYDCYgR3ssjCU70tu4alnGGh0lKx7zp1E eBp9GAvXQf7ziBzMp3r2P1/MkXHXeJADhsy2V5Mv4vqQ2b/nDMfJP0ib2mf2PfoRn4KL Y0r4aZKCNUEne0/8wxI3/DWSa1MT/lGKKdjdDSKbG3PwZd2B8Evau1A+5XQiDG9UVd1b x/Nhssc5xcrLX5z2wxIvOJpUvGnYrifPeMmCjxSLkP8FQpR+TZxi+K7Oo6M1y5gt0BEw EDHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QnmmwIJNP9kkVUAS3cBc0nlBHoiLXYNHoaymUDNzq/A=; b=k69MsfFZr/xHJUiInWf+W42zgNYhko3XBOaAr62kdgoVq8ve7GeDqbD664AUDs3OW8 WceKAc4i1i4afYo3cju3YJjFC+zy+faeOrJMV1OeieKGCQp3ydIbcaQ03vTOWtXvkaiN /EZc+cXM1MofdWAMH/ewU/sDxg7oPGUEsM13YL6te64sW5op3iQDIjt5GNmWBa+n/L4R sLEnTIHnJRaiILnfWPHh86N7WLA0EmXaR+oAhAnlb9GV3bwbgdFM93UiSKbyFhX5lk3/ 1AVBcXQX5sLz14FdQXpytPNqHcQxAnUDqMR1pDYSlgXU0cjZNCtFGBMvWjFHAF9ZPDQD VT2Q== X-Gm-Message-State: APjAAAX2UOzQg25XpjRPHG0SR/Crzx8cEMSgQL3O/DPyDJLtCf0XkrkR JBqqNK36dX8vYeD4BRjo2AA2uQ== X-Google-Smtp-Source: APXvYqzgDYJZq9TRPdA0552l5djc0ZHtNDhcQo2qVJODtZJoCxvM00xf60TYj9RDHxsEBbJOO6yp+w== X-Received: by 2002:a37:48d0:: with SMTP id v199mr745584qka.318.1562803216928; Wed, 10 Jul 2019 17:00:16 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id x8sm1714546qkl.27.2019.07.10.17.00.10 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 10 Jul 2019 17:00:16 -0700 (PDT) From: Matheus Tavares To: Junio C Hamano Cc: git@vger.kernel.org, Thomas Gummerer , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= , Christian Couder , =?utf-8?b?Tmd1eeG7hW4gVGg=?= =?utf-8?b?w6FpIE5n4buNYyBEdXk=?= , =?utf-8?q?SZEDER_G?= =?utf-8?q?=C3=A1bor?= , Olga Telezhnaya , Johannes Schindelin , kernel-usp@googlegroups.com, Michael Haggerty , Ramsay Jones , Daniel Ferreira Subject: [GSoC][PATCH v8 06/10] dir-iterator: add flags parameter to dir_iterator_begin Date: Wed, 10 Jul 2019 20:59:00 -0300 Message-Id: X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add the possibility of giving flags to dir_iterator_begin to initialize a dir-iterator with special options. Currently possible flags are: - DIR_ITERATOR_PEDANTIC, which makes dir_iterator_advance abort immediately in the case of an error, instead of keep looking for the next valid entry; - DIR_ITERATOR_FOLLOW_SYMLINKS, which makes the iterator follow symlinks and include linked directories' contents in the iteration. These new flags will be used in a subsequent patch. Also add tests for the flags' usage and adjust refs/files-backend.c to the new dir_iterator_begin signature. Signed-off-by: Matheus Tavares --- dir-iterator.c | 56 +++++++++++++++++-------- dir-iterator.h | 55 ++++++++++++++++++++----- refs/files-backend.c | 2 +- t/helper/test-dir-iterator.c | 34 +++++++++++---- t/t0066-dir-iterator.sh | 80 ++++++++++++++++++++++++++++++++++++ 5 files changed, 191 insertions(+), 36 deletions(-) diff --git a/dir-iterator.c b/dir-iterator.c index 594fe4d67b..b17e9f970a 100644 --- a/dir-iterator.c +++ b/dir-iterator.c @@ -38,13 +38,16 @@ struct dir_iterator_int { * that will be included in this iteration. */ struct dir_iterator_level *levels; + + /* Combination of flags for this dir-iterator */ + unsigned int flags; }; /* * Push a level in the iter stack and initialize it with information from * the directory pointed by iter->base->path. It is assumed that this * strbuf points to a valid directory path. Return 0 on success and -1 - * otherwise, leaving the stack unchanged. + * otherwise, setting errno accordingly and leaving the stack unchanged. */ static int push_level(struct dir_iterator_int *iter) { @@ -59,11 +62,13 @@ static int push_level(struct dir_iterator_int *iter) level->dir = opendir(iter->base.path.buf); if (!level->dir) { + int saved_errno = errno; if (errno != ENOENT) { warning_errno("error opening directory '%s'", iter->base.path.buf); } iter->levels_nr--; + errno = saved_errno; return -1; } @@ -90,11 +95,13 @@ static int pop_level(struct dir_iterator_int *iter) /* * Populate iter->base with the necessary information on the next iteration * entry, represented by the given dirent de. Return 0 on success and -1 - * otherwise. + * otherwise, setting errno accordingly. */ static int prepare_next_entry_data(struct dir_iterator_int *iter, struct dirent *de) { + int err, saved_errno; + strbuf_addstr(&iter->base.path, de->d_name); /* * We have to reset these because the path strbuf might have @@ -105,13 +112,17 @@ static int prepare_next_entry_data(struct dir_iterator_int *iter, iter->base.basename = iter->base.path.buf + iter->levels[iter->levels_nr - 1].prefix_len; - if (lstat(iter->base.path.buf, &iter->base.st)) { - if (errno != ENOENT) - warning_errno("failed to stat '%s'", iter->base.path.buf); - return -1; - } + if (iter->flags & DIR_ITERATOR_FOLLOW_SYMLINKS) + err = stat(iter->base.path.buf, &iter->base.st); + else + err = lstat(iter->base.path.buf, &iter->base.st); - return 0; + saved_errno = errno; + if (err && errno != ENOENT) + warning_errno("failed to stat '%s'", iter->base.path.buf); + + errno = saved_errno; + return err; } int dir_iterator_advance(struct dir_iterator *dir_iterator) @@ -119,11 +130,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator) struct dir_iterator_int *iter = (struct dir_iterator_int *)dir_iterator; - if (S_ISDIR(iter->base.st.st_mode)) { - if (push_level(iter) && iter->levels_nr == 0) { - /* Pushing the first level failed */ - return dir_iterator_abort(dir_iterator); - } + if (S_ISDIR(iter->base.st.st_mode) && push_level(iter)) { + if (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC) + goto error_out; + if (iter->levels_nr == 0) + goto error_out; } /* Loop until we find an entry that we can give back to the caller. */ @@ -137,22 +148,32 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator) de = readdir(level->dir); if (!de) { - if (errno) + if (errno) { warning_errno("error reading directory '%s'", iter->base.path.buf); - else if (pop_level(iter) == 0) + if (iter->flags & DIR_ITERATOR_PEDANTIC) + goto error_out; + } else if (pop_level(iter) == 0) { return dir_iterator_abort(dir_iterator); + } continue; } if (is_dot_or_dotdot(de->d_name)) continue; - if (prepare_next_entry_data(iter, de)) + if (prepare_next_entry_data(iter, de)) { + if (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC) + goto error_out; continue; + } return ITER_OK; } + +error_out: + dir_iterator_abort(dir_iterator); + return ITER_ERROR; } int dir_iterator_abort(struct dir_iterator *dir_iterator) @@ -178,7 +199,7 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator) return ITER_DONE; } -struct dir_iterator *dir_iterator_begin(const char *path) +struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags) { struct dir_iterator_int *iter = xcalloc(1, sizeof(*iter)); struct dir_iterator *dir_iterator = &iter->base; @@ -189,6 +210,7 @@ struct dir_iterator *dir_iterator_begin(const char *path) ALLOC_GROW(iter->levels, 10, iter->levels_alloc); iter->levels_nr = 0; + iter->flags = flags; /* * Note: stat already checks for NULL or empty strings and diff --git a/dir-iterator.h b/dir-iterator.h index 9b4cb7acd2..08229157c6 100644 --- a/dir-iterator.h +++ b/dir-iterator.h @@ -20,7 +20,8 @@ * A typical iteration looks like this: * * int ok; - * struct dir_iterator *iter = dir_iterator_begin(path); + * unsigned int flags = DIR_ITERATOR_PEDANTIC; + * struct dir_iterator *iter = dir_iterator_begin(path, flags); * * if (!iter) * goto error_handler; @@ -44,6 +45,29 @@ * dir_iterator_advance() again. */ +/* + * Flags for dir_iterator_begin: + * + * - DIR_ITERATOR_PEDANTIC: override dir-iterator's default behavior + * in case of an error at dir_iterator_advance(), which is to keep + * looking for a next valid entry. With this flag, resources are freed + * and ITER_ERROR is returned immediately. In both cases, a meaningful + * warning is emitted. Note: ENOENT errors are always ignored so that + * the API users may remove files during iteration. + * + * - DIR_ITERATOR_FOLLOW_SYMLINKS: make dir-iterator follow symlinks. + * i.e., linked directories' contents will be iterated over and + * iter->base.st will contain information on the referred files, + * not the symlinks themselves, which is the default behavior. Broken + * symlinks are ignored. + * + * Warning: circular symlinks are also followed when + * DIR_ITERATOR_FOLLOW_SYMLINKS is set. The iteration may end up with + * an ELOOP if they happen and DIR_ITERATOR_PEDANTIC is set. + */ +#define DIR_ITERATOR_PEDANTIC (1 << 0) +#define DIR_ITERATOR_FOLLOW_SYMLINKS (1 << 1) + struct dir_iterator { /* The current path: */ struct strbuf path; @@ -58,29 +82,38 @@ struct dir_iterator { /* The current basename: */ const char *basename; - /* The result of calling lstat() on path: */ + /* + * The result of calling lstat() on path; or stat(), if the + * DIR_ITERATOR_FOLLOW_SYMLINKS flag was set at + * dir_iterator's initialization. + */ struct stat st; }; /* - * Start a directory iteration over path. On success, return a - * dir_iterator that holds the internal state of the iteration. - * In case of failure, return NULL and set errno accordingly. + * Start a directory iteration over path with the combination of + * options specified by flags. On success, return a dir_iterator + * that holds the internal state of the iteration. In case of + * failure, return NULL and set errno accordingly. * * The iteration includes all paths under path, not including path * itself and not including "." or ".." entries. * - * path is the starting directory. An internal copy will be made. + * Parameters are: + * - path is the starting directory. An internal copy will be made. + * - flags is a combination of the possible flags to initialize a + * dir-iterator or 0 for default behavior. */ -struct dir_iterator *dir_iterator_begin(const char *path); +struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags); /* * Advance the iterator to the first or next item and return ITER_OK. * If the iteration is exhausted, free the dir_iterator and any - * resources associated with it and return ITER_DONE. On error, free - * dir_iterator and associated resources and return ITER_ERROR. It is - * a bug to use iterator or call this function again after it has - * returned ITER_DONE or ITER_ERROR. + * resources associated with it and return ITER_DONE. + * + * It is a bug to use iterator or call this function again after it + * has returned ITER_DONE or ITER_ERROR (which may be returned iff + * the DIR_ITERATOR_PEDANTIC flag was set). */ int dir_iterator_advance(struct dir_iterator *iterator); diff --git a/refs/files-backend.c b/refs/files-backend.c index 7ed81046d4..b1f8f53a09 100644 --- a/refs/files-backend.c +++ b/refs/files-backend.c @@ -2150,7 +2150,7 @@ static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store, strbuf_addf(&sb, "%s/logs", gitdir); - diter = dir_iterator_begin(sb.buf); + diter = dir_iterator_begin(sb.buf, 0); if(!diter) return empty_ref_iterator_begin(); diff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c index fab1ff6237..a5b96cb0dc 100644 --- a/t/helper/test-dir-iterator.c +++ b/t/helper/test-dir-iterator.c @@ -4,29 +4,44 @@ #include "iterator.h" #include "dir-iterator.h" -/* Argument is a directory path to iterate over */ +/* + * usage: + * tool-test dir-iterator [--follow-symlinks] [--pedantic] directory_path + */ int cmd__dir_iterator(int argc, const char **argv) { struct strbuf path = STRBUF_INIT; struct dir_iterator *diter; + unsigned int flags = 0; + int iter_status; + + for (++argv, --argc; *argv && starts_with(*argv, "--"); ++argv, --argc) { + if (strcmp(*argv, "--follow-symlinks") == 0) + flags |= DIR_ITERATOR_FOLLOW_SYMLINKS; + else if (strcmp(*argv, "--pedantic") == 0) + flags |= DIR_ITERATOR_PEDANTIC; + else + die("invalid option '%s'", *argv); + } - if (argc < 2) - die("BUG: test-dir-iterator needs one argument"); - - strbuf_add(&path, argv[1], strlen(argv[1])); + if (!*argv || argc != 1) + die("dir-iterator needs exactly one non-option argument"); - diter = dir_iterator_begin(path.buf); + strbuf_add(&path, *argv, strlen(*argv)); + diter = dir_iterator_begin(path.buf, flags); if (!diter) { printf("dir_iterator_begin failure: %d\n", errno); exit(EXIT_FAILURE); } - while (dir_iterator_advance(diter) == ITER_OK) { + while ((iter_status = dir_iterator_advance(diter)) == ITER_OK) { if (S_ISDIR(diter->st.st_mode)) printf("[d] "); else if (S_ISREG(diter->st.st_mode)) printf("[f] "); + else if (S_ISLNK(diter->st.st_mode)) + printf("[s] "); else printf("[?] "); @@ -34,5 +49,10 @@ int cmd__dir_iterator(int argc, const char **argv) diter->path.buf); } + if (iter_status != ITER_DONE) { + printf("dir_iterator_advance failure\n"); + return 1; + } + return 0; } diff --git a/t/t0066-dir-iterator.sh b/t/t0066-dir-iterator.sh index cc4b19c34c..9354d3f1ed 100755 --- a/t/t0066-dir-iterator.sh +++ b/t/t0066-dir-iterator.sh @@ -65,4 +65,84 @@ test_expect_success 'begin should fail upon non directory paths' ' test_cmp expected-non-dir-output actual-non-dir-output ' +test_expect_success POSIXPERM,SANITY 'advance should not fail on errors by default' ' + cat >expected-no-permissions-output <<-EOF && + [d] (a) [a] ./dir3/a + EOF + + mkdir -p dir3/a && + >dir3/a/b && + chmod 0 dir3/a && + + test-tool dir-iterator ./dir3 >actual-no-permissions-output && + test_cmp expected-no-permissions-output actual-no-permissions-output && + chmod 755 dir3/a && + rm -rf dir3 +' + +test_expect_success POSIXPERM,SANITY 'advance should fail on errors, w/ pedantic flag' ' + cat >expected-no-permissions-pedantic-output <<-EOF && + [d] (a) [a] ./dir3/a + dir_iterator_advance failure + EOF + + mkdir -p dir3/a && + >dir3/a/b && + chmod 0 dir3/a && + + test_must_fail test-tool dir-iterator --pedantic ./dir3 \ + >actual-no-permissions-pedantic-output && + test_cmp expected-no-permissions-pedantic-output \ + actual-no-permissions-pedantic-output && + chmod 755 dir3/a && + rm -rf dir3 +' + +test_expect_success SYMLINKS 'setup dirs with symlinks' ' + mkdir -p dir4/a && + mkdir -p dir4/b/c && + >dir4/a/d && + ln -s d dir4/a/e && + ln -s ../b dir4/a/f && + + mkdir -p dir5/a/b && + mkdir -p dir5/a/c && + ln -s ../c dir5/a/b/d && + ln -s ../ dir5/a/b/e && + ln -s ../../ dir5/a/b/f +' + +test_expect_success SYMLINKS 'dir-iterator should not follow symlinks by default' ' + cat >expected-no-follow-sorted-output <<-EOF && + [d] (a) [a] ./dir4/a + [d] (b) [b] ./dir4/b + [d] (b/c) [c] ./dir4/b/c + [f] (a/d) [d] ./dir4/a/d + [s] (a/e) [e] ./dir4/a/e + [s] (a/f) [f] ./dir4/a/f + EOF + + test-tool dir-iterator ./dir4 >out && + sort out >actual-no-follow-sorted-output && + + test_cmp expected-no-follow-sorted-output actual-no-follow-sorted-output +' + +test_expect_success SYMLINKS 'dir-iterator should follow symlinks w/ follow flag' ' + cat >expected-follow-sorted-output <<-EOF && + [d] (a) [a] ./dir4/a + [d] (a/f) [f] ./dir4/a/f + [d] (a/f/c) [c] ./dir4/a/f/c + [d] (b) [b] ./dir4/b + [d] (b/c) [c] ./dir4/b/c + [f] (a/d) [d] ./dir4/a/d + [f] (a/e) [e] ./dir4/a/e + EOF + + test-tool dir-iterator --follow-symlinks ./dir4 >out && + sort out >actual-follow-sorted-output && + + test_cmp expected-follow-sorted-output actual-follow-sorted-output +' + test_done From patchwork Wed Jul 10 23:59:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11039109 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7FB0C6C5 for ; Thu, 11 Jul 2019 00:00:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 718D128A00 for ; Thu, 11 Jul 2019 00:00:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6539528A09; Thu, 11 Jul 2019 00:00:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0193628A00 for ; Thu, 11 Jul 2019 00:00:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727801AbfGKAAX (ORCPT ); Wed, 10 Jul 2019 20:00:23 -0400 Received: from mail-qk1-f193.google.com ([209.85.222.193]:33515 "EHLO mail-qk1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727386AbfGKAAW (ORCPT ); Wed, 10 Jul 2019 20:00:22 -0400 Received: by mail-qk1-f193.google.com with SMTP id r6so3428574qkc.0 for ; Wed, 10 Jul 2019 17:00:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eaYJwpJwCwlCFXJ34eCSRh6aqVIv7haGmVVvxs7jUKA=; b=zVk8ZLgMl0KPWoFBU2Zy0KDHo3tQI58k8ydeEAgIQEH9vfZBT6PmTR6Diz3O46+3Ov Jo6LiAww63sxCy3wazcnyumxM7KdCKn0Fvc3odvtVBnj2dVWV+bhUoM0Mzeyc5V3CO7L Z8Md+OqPkJ2OLCK7WuLaBfX6KxaCcRdq+BIH6c2PCiJQk1QeGhuHdzEhkTBDs7NaeMml a998VBuJeIaaXGqQ2/r978l4vZseymOdi0on3kYfzxEi4wwvKITtbRXXgtld/kt4PTx/ BJem2vZvTK3A8EieDGKgHPdZME1UPTUaMBBRu/89JAnwoYd1lXYN+hlmMdysUdmS10Ia 6JRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eaYJwpJwCwlCFXJ34eCSRh6aqVIv7haGmVVvxs7jUKA=; b=Qd7FSqd1FlbWUwjm51uBaHGKKbsH4R1Jg3GmhfJn1D1XRmLYK8IEuusOp8w5OGnsPu m8GQCsFmW8Z0VF+FRvRrS/XTppfja0NP5eVs7xE2Yi0CMEzACWzJqmhd5JA74XTpH6aF qFyAY+FQlt1CuG4CJ8tZ3CAcfOeALeBMT7X9HdAM31SFFnPW3mFMaVc/kP98LyCqwKV0 xBJbWw0sD/XgK+Ohx9atFZBO5zfbvGu3VcmKi6fbOhpIzhxpFpU7+uXJjIs1uyus6EZU R5nDtCjeBzfX6noVIHh/HX1wE5VvpSuIDsHwAxWgQlJHwU1Dhm21rbnH+3ZlIfHflpq8 QvXw== X-Gm-Message-State: APjAAAUkoESMU8/yACsD/GDtM9ooIa0ZQjuC7wgQLzzcy3ctLfJf5tJE TaxTgDQUIHvrKBecR+nuBvP1qQ== X-Google-Smtp-Source: APXvYqz6k1u8MsiSvd4FYoN8H1vWZ8bCI5hnZ69vWKZHjrpaYoe7oR8P61hFWamkHSaxWSY/9xf6kA== X-Received: by 2002:a37:bf07:: with SMTP id p7mr683051qkf.315.1562803221660; Wed, 10 Jul 2019 17:00:21 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id x8sm1714546qkl.27.2019.07.10.17.00.17 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 10 Jul 2019 17:00:21 -0700 (PDT) From: Matheus Tavares To: Junio C Hamano Cc: git@vger.kernel.org, Thomas Gummerer , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= , Christian Couder , =?utf-8?b?Tmd1eeG7hW4gVGg=?= =?utf-8?b?w6FpIE5n4buNYyBEdXk=?= , =?utf-8?q?SZEDER_G?= =?utf-8?q?=C3=A1bor?= , Olga Telezhnaya , Johannes Schindelin , kernel-usp@googlegroups.com Subject: [GSoC][PATCH v8 07/10] clone: copy hidden paths at local clone Date: Wed, 10 Jul 2019 20:59:01 -0300 Message-Id: <3da6408e045de6e39166227a60472bd1952664ad.1562801255.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Make the copy_or_link_directory function no longer skip hidden directories. This function, used to copy .git/objects, currently skips all hidden directories but not hidden files, which is an odd behaviour. The reason for that could be unintentional: probably the intention was to skip '.' and '..' only but it ended up accidentally skipping all directories starting with '.'. Besides being more natural, the new behaviour is more permissive to the user. Also adjust tests to reflect this behaviour change. Signed-off-by: Matheus Tavares Signed-off-by: Ævar Arnfjörð Bjarmason Co-authored-by: Ævar Arnfjörð Bjarmason --- builtin/clone.c | 2 +- t/t5604-clone-reference.sh | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/builtin/clone.c b/builtin/clone.c index 4a0a2455a7..9dd083e34d 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -430,7 +430,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest, continue; } if (S_ISDIR(buf.st_mode)) { - if (de->d_name[0] != '.') + if (!is_dot_or_dotdot(de->d_name)) copy_or_link_directory(src, dest, src_repo, src_baselen); continue; diff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh index 459ad8a20b..4894237ab8 100755 --- a/t/t5604-clone-reference.sh +++ b/t/t5604-clone-reference.sh @@ -247,16 +247,25 @@ test_expect_success 'clone a repo with garbage in objects/*' ' done && find S-* -name "*some*" | sort >actual && cat >expected <<-EOF && + S--dissociate/.git/objects/.some-hidden-dir + S--dissociate/.git/objects/.some-hidden-dir/.some-dot-file + S--dissociate/.git/objects/.some-hidden-dir/some-file S--dissociate/.git/objects/.some-hidden-file S--dissociate/.git/objects/some-dir S--dissociate/.git/objects/some-dir/.some-dot-file S--dissociate/.git/objects/some-dir/some-file S--dissociate/.git/objects/some-file + S--local/.git/objects/.some-hidden-dir + S--local/.git/objects/.some-hidden-dir/.some-dot-file + S--local/.git/objects/.some-hidden-dir/some-file S--local/.git/objects/.some-hidden-file S--local/.git/objects/some-dir S--local/.git/objects/some-dir/.some-dot-file S--local/.git/objects/some-dir/some-file S--local/.git/objects/some-file + S--no-hardlinks/.git/objects/.some-hidden-dir + S--no-hardlinks/.git/objects/.some-hidden-dir/.some-dot-file + S--no-hardlinks/.git/objects/.some-hidden-dir/some-file S--no-hardlinks/.git/objects/.some-hidden-file S--no-hardlinks/.git/objects/some-dir S--no-hardlinks/.git/objects/some-dir/.some-dot-file From patchwork Wed Jul 10 23:59:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11039111 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4039D6C5 for ; Thu, 11 Jul 2019 00:00:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 301F528A00 for ; Thu, 11 Jul 2019 00:00:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 23F3128A09; Thu, 11 Jul 2019 00:00:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BF68828A00 for ; Thu, 11 Jul 2019 00:00:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727812AbfGKAA1 (ORCPT ); Wed, 10 Jul 2019 20:00:27 -0400 Received: from mail-qt1-f195.google.com ([209.85.160.195]:32808 "EHLO mail-qt1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727386AbfGKAA1 (ORCPT ); Wed, 10 Jul 2019 20:00:27 -0400 Received: by mail-qt1-f195.google.com with SMTP id r6so282008qtt.0 for ; Wed, 10 Jul 2019 17:00:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DPoy5SCr9H6wbbntCXImFSyMCGfsAC0sldlgcwIwbqI=; b=GMeilZ7P8+a3ZKH6C3/OXcGyd95DHBHbLrNf5TOZz93dqjoB/ARPIfRFKYUrwWsy8s 2iVxLNyabrL3tahK5jXk1UT7abcZvtcJpLCs9kOWPFubuPaDrrE4+PcuHaWhi8YLqdFC h+lwuxHSmYP+HAi6ewcN4g7mRGBffbXJU0CUJy1/FfXpklMDLDxtFDjMeKGQGxn4V6/6 FnBHC9ZhjQlMmtB7ZaYYfb/AtmlaWNAp8EI/kNohwpZVPqBrSCyXcOfZ9XM2Gp9uiUDa 8xWRz09iLCVXR6/a7IoDTJNfaY8bLTap5+9DKQxm89MgmiOgMfpMtwXiElIqBJ9+NVh+ AtZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DPoy5SCr9H6wbbntCXImFSyMCGfsAC0sldlgcwIwbqI=; b=XeX2Ig07ajBDkkh44rRB7HwiaFqMGEDR7QVUozDyEbWhpGzBiOzpwN2f+t/nZTe5ap uh/mn1lpbAa4FUmOwe65Xxo0etUZbvivKMksens6hoSnlKhUDyHsxZwjV06zNVfBDtkL AJgZmmPVeI/zaoGoBzW/PCQfY7zNONHQ0bAa/WrMEyOfZ32XzrkGJSPysQQUadePPAa4 rev4jnFzaJ6vgesMdbS9APGfRxFa6h7q99IcI9O9w6uJb0e1gSg/lf7grh6LRT+yOETb hqV5Hu5arbzOXlw2BvxRFhnuPiZ6ST+58UtZJOnXeO/Kbq1eXJZ00CusbywBSyrF5/uc DOeQ== X-Gm-Message-State: APjAAAWdysLk9Ao4tg9pySvPYypJXt4uZzFyBGvHOsjvECtvlj2ua1wG 5wnbPBLYovZtDXzIR2kDxETgXA== X-Google-Smtp-Source: APXvYqzmBdKMRx/F/+RQrouxUPTaIWVvU2Ee8LBvq3m16EJ2AbqeKsqJtZl2QDzuy2FbQkhs1Kcdbw== X-Received: by 2002:ac8:2b01:: with SMTP id 1mr601262qtu.177.1562803226491; Wed, 10 Jul 2019 17:00:26 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id x8sm1714546qkl.27.2019.07.10.17.00.22 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 10 Jul 2019 17:00:26 -0700 (PDT) From: Matheus Tavares To: Junio C Hamano Cc: git@vger.kernel.org, Thomas Gummerer , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= , Christian Couder , =?utf-8?b?Tmd1eeG7hW4gVGg=?= =?utf-8?b?w6FpIE5n4buNYyBEdXk=?= , =?utf-8?q?SZEDER_G?= =?utf-8?q?=C3=A1bor?= , Olga Telezhnaya , Johannes Schindelin , kernel-usp@googlegroups.com Subject: [GSoC][PATCH v8 08/10] clone: extract function from copy_or_link_directory Date: Wed, 10 Jul 2019 20:59:02 -0300 Message-Id: X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Extract dir creation code snippet from copy_or_link_directory to its own function named mkdir_if_missing. This change will help to remove copy_or_link_directory's explicit recursion, which will be done in a following patch. Also makes the code more readable. Signed-off-by: Matheus Tavares --- builtin/clone.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 9dd083e34d..96566c1bab 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -394,6 +394,21 @@ static void copy_alternates(struct strbuf *src, const char *src_repo) fclose(in); } +static void mkdir_if_missing(const char *pathname, mode_t mode) +{ + struct stat st; + + if (!mkdir(pathname, mode)) + return; + + if (errno != EEXIST) + die_errno(_("failed to create directory '%s'"), pathname); + else if (stat(pathname, &st)) + die_errno(_("failed to stat '%s'"), pathname); + else if (!S_ISDIR(st.st_mode)) + die(_("%s exists and is not a directory"), pathname); +} + static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest, const char *src_repo, int src_baselen) { @@ -406,14 +421,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest, if (!dir) die_errno(_("failed to open '%s'"), src->buf); - if (mkdir(dest->buf, 0777)) { - if (errno != EEXIST) - die_errno(_("failed to create directory '%s'"), dest->buf); - else if (stat(dest->buf, &buf)) - die_errno(_("failed to stat '%s'"), dest->buf); - else if (!S_ISDIR(buf.st_mode)) - die(_("%s exists and is not a directory"), dest->buf); - } + mkdir_if_missing(dest->buf, 0777); strbuf_addch(src, '/'); src_len = src->len; From patchwork Wed Jul 10 23:59:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11039113 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D011C6C5 for ; Thu, 11 Jul 2019 00:00:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AE9642883A for ; Thu, 11 Jul 2019 00:00:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7725328A22; Thu, 11 Jul 2019 00:00:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A23CD28984 for ; Thu, 11 Jul 2019 00:00:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727831AbfGKAAd (ORCPT ); Wed, 10 Jul 2019 20:00:33 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:35670 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727386AbfGKAAd (ORCPT ); Wed, 10 Jul 2019 20:00:33 -0400 Received: by mail-qt1-f196.google.com with SMTP id d23so4493622qto.2 for ; Wed, 10 Jul 2019 17:00:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jVkq6cenj1ZapQF1Fr1X5AkjZwd58aekWv74Ku8RjIg=; b=LOUWGg9uNKDBH9dPxD2wn5q+COAEWdZ7SX3Eo3mlypm7dzoso1pnooIowj7v83aLQt ZYHucljH1CSAl2/ToLOwiub7wKo30VJ8utX+z0ER5v5gmpt/Wi4MTGQ78EVZylxuqxeF Un1MsrLOosOEhGnasDqu+JcjonlXhu/z8c9utEOyfaWHG35WWhhMRiS580h1MCERe7xf JCGedmmzScaSLSKg1gOMeqPBMrsAtz7yuYfcN5yfrB8kxU9WkgExnF7+emiD36J5gelr MLwOBGHBUnuBn2T18OsfEn1xNZIVb9HMW7o+sdegI/v4PiUBUTeqCuW2JP03Hl86oqIe IMJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jVkq6cenj1ZapQF1Fr1X5AkjZwd58aekWv74Ku8RjIg=; b=JDNhC3Q5WZbnUzhOtC/XScnVJARJa7wC0k6f2W35PID27gjPQvI8e8+aCgnfjZi+t2 7X4VOJgzd8RRBm+XrLWlTpNv3VM/bQ14Yh8bGr9y4ej2+7bxFz5B8QOGBhohMZuCXHiZ tzWmY7h+ou40Ycx25pYoKYn2u8v793D6ftV+91UvAPlFby6d7hNWF6aKIQGT1lqX3cBd Las7uPBt/8fmg7j9Up/3SZYmkcJ4wTcFDghx6zHLTN6WSqSWjZFkUNUcC0WnZ+ANO32N LLsZ9WqIpZ+qlezjkMVS8qlrMJ19T2vpms5Vz1lgFkU70pD3EmD5mzbBLbw6PQ0q9ooi gUIA== X-Gm-Message-State: APjAAAW8oDg7V5znFPBeMiwGhjOHisouX4iiXeXpyorxx9g+slIjzVxJ RVEuu5Ld2z/carR58Cz/HTixRQ== X-Google-Smtp-Source: APXvYqwN0q1zSg99W/Bf/lmFltIEddz628AWsSUEhb/+tOpZnAdLlMRyBrQthkTWKtnkNdQyPLVpMA== X-Received: by 2002:ac8:30d2:: with SMTP id w18mr624948qta.296.1562803231734; Wed, 10 Jul 2019 17:00:31 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id x8sm1714546qkl.27.2019.07.10.17.00.26 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 10 Jul 2019 17:00:31 -0700 (PDT) From: Matheus Tavares To: Junio C Hamano Cc: git@vger.kernel.org, Thomas Gummerer , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= , Christian Couder , =?utf-8?b?Tmd1eeG7hW4gVGg=?= =?utf-8?b?w6FpIE5n4buNYyBEdXk=?= , =?utf-8?q?SZEDER_G?= =?utf-8?q?=C3=A1bor?= , Olga Telezhnaya , Johannes Schindelin , kernel-usp@googlegroups.com, Jeff King Subject: [GSoC][PATCH v8 09/10] clone: use dir-iterator to avoid explicit dir traversal Date: Wed, 10 Jul 2019 20:59:03 -0300 Message-Id: X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Replace usage of opendir/readdir/closedir API to traverse directories recursively, at copy_or_link_directory function, by the dir-iterator API. This simplifies the code and avoids recursive calls to copy_or_link_directory. This process also makes copy_or_link_directory call die() in case of an error on readdir or stat inside dir_iterator_advance. Previously it would just print a warning for errors on stat and ignore errors on readdir, which isn't nice because a local git clone could succeed even though the .git/objects copy didn't fully succeed. Signed-off-by: Matheus Tavares --- builtin/clone.c | 47 +++++++++++++++++++++++++---------------------- 1 file changed, 25 insertions(+), 22 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 96566c1bab..47cb4a2a8e 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -23,6 +23,8 @@ #include "transport.h" #include "strbuf.h" #include "dir.h" +#include "dir-iterator.h" +#include "iterator.h" #include "sigchain.h" #include "branch.h" #include "remote.h" @@ -410,42 +412,39 @@ static void mkdir_if_missing(const char *pathname, mode_t mode) } static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest, - const char *src_repo, int src_baselen) + const char *src_repo) { - struct dirent *de; - struct stat buf; int src_len, dest_len; - DIR *dir; - - dir = opendir(src->buf); - if (!dir) - die_errno(_("failed to open '%s'"), src->buf); + struct dir_iterator *iter; + int iter_status; + unsigned int flags; mkdir_if_missing(dest->buf, 0777); + flags = DIR_ITERATOR_PEDANTIC | DIR_ITERATOR_FOLLOW_SYMLINKS; + iter = dir_iterator_begin(src->buf, flags); + + if (!iter) + die_errno(_("failed to start iterator over '%s'"), src->buf); + strbuf_addch(src, '/'); src_len = src->len; strbuf_addch(dest, '/'); dest_len = dest->len; - while ((de = readdir(dir)) != NULL) { + while ((iter_status = dir_iterator_advance(iter)) == ITER_OK) { strbuf_setlen(src, src_len); - strbuf_addstr(src, de->d_name); + strbuf_addstr(src, iter->relative_path); strbuf_setlen(dest, dest_len); - strbuf_addstr(dest, de->d_name); - if (stat(src->buf, &buf)) { - warning (_("failed to stat %s\n"), src->buf); - continue; - } - if (S_ISDIR(buf.st_mode)) { - if (!is_dot_or_dotdot(de->d_name)) - copy_or_link_directory(src, dest, - src_repo, src_baselen); + strbuf_addstr(dest, iter->relative_path); + + if (S_ISDIR(iter->st.st_mode)) { + mkdir_if_missing(dest->buf, 0777); continue; } /* Files that cannot be copied bit-for-bit... */ - if (!strcmp(src->buf + src_baselen, "/info/alternates")) { + if (!strcmp(iter->relative_path, "info/alternates")) { copy_alternates(src, src_repo); continue; } @@ -462,7 +461,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest, if (copy_file_with_time(dest->buf, src->buf, 0666)) die_errno(_("failed to copy file to '%s'"), dest->buf); } - closedir(dir); + + if (iter_status != ITER_DONE) { + strbuf_setlen(src, src_len); + die(_("failed to iterate over '%s'"), src->buf); + } } static void clone_local(const char *src_repo, const char *dest_repo) @@ -480,7 +483,7 @@ static void clone_local(const char *src_repo, const char *dest_repo) get_common_dir(&dest, dest_repo); strbuf_addstr(&src, "/objects"); strbuf_addstr(&dest, "/objects"); - copy_or_link_directory(&src, &dest, src_repo, src.len); + copy_or_link_directory(&src, &dest, src_repo); strbuf_release(&src); strbuf_release(&dest); } From patchwork Wed Jul 10 23:59:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11039115 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 65EAA6C5 for ; Thu, 11 Jul 2019 00:00:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58B00288F3 for ; Thu, 11 Jul 2019 00:00:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 569D428A3C; Thu, 11 Jul 2019 00:00:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 03ECD28A0F for ; Thu, 11 Jul 2019 00:00:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727844AbfGKAAj (ORCPT ); Wed, 10 Jul 2019 20:00:39 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:38952 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727386AbfGKAAi (ORCPT ); Wed, 10 Jul 2019 20:00:38 -0400 Received: by mail-qt1-f194.google.com with SMTP id l9so4467383qtu.6 for ; Wed, 10 Jul 2019 17:00:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TA7qLhwVjgT7/njeUhcABACRPDG7IyBV0uSnJgIka+0=; b=lYz1Vvn1ABgdlgooOpEjaqM0BTPoM/K66YIOgC0TWF5GpNQs67s0wjNmbfQjsYVBNi amwvVA3BicggBGSh7WLJdBRqdwfq6E/3ol72Yn4BbXFJI+1QFg2OhdCRqC9YYb3GUhGm Bq31njUI18j9Vm2vDb1Z0PSN2PoZrX5NGSzQ1Ns76kh7+JKYDpQ58hI0zrpIzifgu7IH 3lWLbTRuT+NJY0UqIknPXx6DwwbU61mZIhxi3lEkqp9QwCHUZFVCLCgj4urigwKOkJrO nsZrEPDXS7+0SyC9bqU80m6izoVp6QCRAbVLJfhtZXhzTtvnWVy1xiuu9pLI386ab0Xh oW6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TA7qLhwVjgT7/njeUhcABACRPDG7IyBV0uSnJgIka+0=; b=XWI2duvhnL3WnJWoFPctPoJQQZHk8vDozppIzVSUpQMNoxhwJrlZGPIaMtYkfM8InY pVlCGVSjff+nktx3/8NS2FSaG1yN72LwV4HTY19Pj6dvNYwSDYeZRdKPYM3x5URqHGVH tAjR7BJseGSABKfR2WhcuNge4uWaecur5jvFrMZ9F3gVHMrWt0V1yg5vRl/9TwtYNFvU MSLw3IMv4BHz+6JA7fbf8HqsG89nHs20xX4R++DuyHFNGTYG2dXcVcAPqghLlTEkL0RE AHLHDP12X0gHuZm93H9QocE5dgQ+DnqikRmxmFsS2zZU/E2g8E0fm1nMJHOLhPmBGCoL GqHw== X-Gm-Message-State: APjAAAVYEulHhPlC0ch5Yt7btvFsqdu4LpQMoZm1ix5E3hI9dSJWeORa XPydzgpBLz59mzyvgMicelaAAA== X-Google-Smtp-Source: APXvYqyRrRbPKjhKn0qhq6l47wNGeLxYn7BYr93KT7HT8M86MNWvXax/yghxrnx0g0JwgFKQIiGS1w== X-Received: by 2002:ac8:359a:: with SMTP id k26mr578155qtb.87.1562803237774; Wed, 10 Jul 2019 17:00:37 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id x8sm1714546qkl.27.2019.07.10.17.00.32 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 10 Jul 2019 17:00:37 -0700 (PDT) From: Matheus Tavares To: Junio C Hamano Cc: git@vger.kernel.org, Thomas Gummerer , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= , Christian Couder , =?utf-8?b?Tmd1eeG7hW4gVGg=?= =?utf-8?b?w6FpIE5n4buNYyBEdXk=?= , =?utf-8?q?SZEDER_G?= =?utf-8?q?=C3=A1bor?= , Olga Telezhnaya , Johannes Schindelin , kernel-usp@googlegroups.com, Jeff King Subject: [GSoC][PATCH v8 10/10] clone: replace strcmp by fspathcmp Date: Wed, 10 Jul 2019 20:59:04 -0300 Message-Id: <782ca07eed2c9bac4378e5128ff996b25ed86a43.1562801255.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Replace the use of strcmp by fspathcmp at copy_or_link_directory, which is more permissive/friendly to case-insensitive file systems. Suggested-by: Nguyễn Thái Ngọc Duy Signed-off-by: Matheus Tavares --- builtin/clone.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builtin/clone.c b/builtin/clone.c index 47cb4a2a8e..8da696ef30 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -444,7 +444,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest, } /* Files that cannot be copied bit-for-bit... */ - if (!strcmp(iter->relative_path, "info/alternates")) { + if (!fspathcmp(iter->relative_path, "info/alternates")) { copy_alternates(src, src_repo); continue; }