From patchwork Tue Dec 12 14:16:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Ondrej Pohorelsky X-Patchwork-Id: 13489306 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="HfZl28nT" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B99DD8F for ; Tue, 12 Dec 2023 06:16:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702390600; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=tms1BQbG6Qr9dCd2X8ma5kxr0csRIl22e/3k7Y/7AYU=; b=HfZl28nTG+v6p1hQ0eGRhTaP47OW6cti16+XBFlmODdNIFNZW1prrFy6Pyh64PjQf4qd1G Q6KeZ/cBt518Fna03ByXNFxnfN61vTD4xkSDWCZlCrw1U7Da1esfgFcnbQPN7XCqjFnAlG wAnaxq5fsDr5gd+Y1MIFAHidL+qirOU= Received: from mail-ej1-f71.google.com (mail-ej1-f71.google.com [209.85.218.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-247-8xnm4PCrMqCcZb7D1vft0Q-1; Tue, 12 Dec 2023 09:16:38 -0500 X-MC-Unique: 8xnm4PCrMqCcZb7D1vft0Q-1 Received: by mail-ej1-f71.google.com with SMTP id a640c23a62f3a-a1bd7a5931eso314303266b.3 for ; Tue, 12 Dec 2023 06:16:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702390597; x=1702995397; h=content-transfer-encoding:to:subject:message-id:date:from :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=tms1BQbG6Qr9dCd2X8ma5kxr0csRIl22e/3k7Y/7AYU=; b=LcuBI4CH7OnFovs22dqZjOH0I0Yxw+gbFPL8zgLQ21CtFlbOH+XWb+yBXuWNaceBJc iR+rbuh8LjDtuNLa75fApNQp+tov6q6Ms3m9wP+tY7vcywsv0ZMTnBl3NAJzm+dKunsm FOO0BJiLIxwh2gnzuEwBen0M4e05ox+VQK9Ed662qm/ISab1TVqkFaprqeJrrKw6CNhB EF8d91/xc7bjVU550eVtbHM4qW6ozuYVgaDjH1lvDWS9aFWs+JULceUjtRTsBPdfSZsj OSIfT++3JNUIxt7J8NbBiQJR32eRuBLMi4vrUiJlVQE5+/vOlvtELfrvgVEyNxnLUEVZ zTDg== X-Gm-Message-State: AOJu0YwSZ+VE5ms2kxEab0G1v2gb8IhYDKHXY6BO3DJkSZo7NOtLzO2w uBZMIeC1muT/kjId6YKAXjgQtAtZ3o0LJEOJOMxC5dYTGwtkEDh2mqUpQ/EhWj/WUH0om8uKkF2 D734FRFJsedMhWkFagLKVh2ijOtCnN/+yRZBF7x8= X-Received: by 2002:a17:907:6d1d:b0:a1e:eecd:6f88 with SMTP id sa29-20020a1709076d1d00b00a1eeecd6f88mr3902252ejc.77.1702390596989; Tue, 12 Dec 2023 06:16:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IEYDrBbQBV/AaxSfLg+Di3LLjBAcyJPcZQ/Vgt9UWqMuLPhI+cVxF+mBIos9Mw7jzq1G/AQje8Lt0bkSK4BaEk= X-Received: by 2002:a17:907:6d1d:b0:a1e:eecd:6f88 with SMTP id sa29-20020a1709076d1d00b00a1eeecd6f88mr3902243ejc.77.1702390596689; Tue, 12 Dec 2023 06:16:36 -0800 (PST) Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Ondrej Pohorelsky Date: Tue, 12 Dec 2023 15:16:26 +0100 Message-ID: Subject: Test breakage with zlib-ng To: git@vger.kernel.org Hi everyone, As some might have heard, there is a proposal for Fedora 40 to transition from zlib to zlib-ng[0]. Because of this, there has been a rebuild of all packages to ensure every package works under zlib-ng. Git test suit has three breakages in t6300-for-each-ref.sh. To be precise, it is: not ok 35 - basic atom: refs/heads/main objectsize:disk not ok 107 - basic atom: refs/tags/testtag objectsize:disk not ok 108 - basic atom: refs/tags/testtag *objectsize:disk All of these tests are atomic, and they compare the result against $disklen. I discussed it with Tulio Magno Quites Machado Filho, who ran the tests and is owner of the proposal. It seems like the compression of zlib-ng is shaving/adding some bytes to the actual output, which then fails the comparison. Here is an example: ``` expecting success of 6300.35 'basic atom: refs/heads/main objectsize:disk': git for-each-ref --format="%($format)" "$ref" >actual && sanitize_pgp actual.clean && test_cmp expected actual.clean ++ git for-each-ref '--format=%(objectsize:disk)' refs/heads/main ++ sanitize_pgp ++ perl -ne ' /^-----END PGP/ and $in_pgp = 0; print unless $in_pgp; /^-----BEGIN PGP/ and $in_pgp = 1; ' ++ command /usr/bin/perl -ne ' /^-----END PGP/ and $in_pgp = 0; print unless $in_pgp; /^-----BEGIN PGP/ and $in_pgp = 1; ' ++ test_cmp expected actual.clean ++ test 2 -ne 2 ++ eval 'diff -u' '"$@"' +++ diff -u expected actual.clean error: last command exited with $?=1 not ok 35 - basic atom: refs/heads/main objectsize:disk ``` The whole build log can be found here[1]. I can easily patch these tests in Fedora to be compatible with zlib-ng only by not comparing to $disklen, but a concrete value, however I would like to have a universal solution that works with both zlib and zlib-ng. So if anyone has an idea on how to do it, please let me know. Thanks [0]https://discussion.fedoraproject.org/t/f40-change-proposal-transitioning-to-zlib-ng-as-a-compatible-replacement-for-zlib-system-wide/95807 [1]https://download.copr.fedorainfracloud.org/results/tuliom/zlib-ng-compat-mpb/fedora-rawhide-x86_64/06729801-git/builder-live.log.gz Cheers, Ondřej Pohořelský Reported-by: Ondrej Pohorelsky Signed-off-by: René Scharfe --- expected 2023-12-06 21:06:07.808849497 +0000 +++ actual.clean 2023-12-06 21:06:07.812849541 +0000 @@ -1 +1 @@ -138 +137 From patchwork Wed Dec 13 12:28:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ren=C3=A9_Scharfe?= X-Patchwork-Id: 13490841 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=web.de header.i=l.s.r@web.de header.b="gExMqnPu" Received: from mout.web.de (mout.web.de [212.227.17.11]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6628093 for ; Wed, 13 Dec 2023 04:29:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=s29768273; t=1702470537; x=1703075337; i=l.s.r@web.de; bh=jYuPLFiOkaITAqk9bMbolbLebKOah42wCYSxa1bjkIE=; h=X-UI-Sender-Class:Date:Subject:From:To:Cc:References: In-Reply-To; b=gExMqnPuslj36SWxnNSj4FEbDMdU9mfbycjjL/ZoO5pdySmrOGOnP5d2LEQR3iN9 4l1dKf82Xk7KxFATt5VLyGuFFMW4hxoB+UZ6RZVwglZ/764T2W0fhaWvJiT/60/0l MVjT5C18VdmwgbQzaxo5eMxAukq2vA38RPpkYhF/A/wi02aJOa0UD/DgGSAVmXIpQ G6yvL7ykkmUNjtSoHBze6qgT+zrTKb3k8eDwAUNfHm1vU1fk8EggecT6Rz1sIcBg6 YX9791N+E23pvNbd3A80jheXCakx3ecY65L3UOn5E+0uj/Jw/evpSFyHYsscnlp7a SpH/AeebqcNJAwQrwg== X-UI-Sender-Class: 814a7b36-bfc1-4dae-8640-3722d8ec6cd6 Received: from [192.168.178.29] ([79.203.29.38]) by smtp.web.de (mrweb106 [213.165.67.124]) with ESMTPSA (Nemesis) id 1MgANA-1rhWul2Ssg-00hpja; Wed, 13 Dec 2023 13:28:57 +0100 Message-ID: <65557f2d-9de0-49ae-a858-80476aa52b68@web.de> Date: Wed, 13 Dec 2023 13:28:56 +0100 Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH 2/1] test-lib-functions: add object size functions From: =?utf-8?q?Ren=C3=A9_Scharfe?= To: Jeff King , git@vger.kernel.org Cc: Ondrej Pohorelsky , "brian m . carlson" , Junio C Hamano References: <9feeb6cf-aabf-4002-917f-3f6c27547bc8@web.de> <20231212200153.GB1127366@coredump.intra.peff.net> Content-Language: en-US In-Reply-To: X-Provags-ID: V03:K1:uFFhX1jbY3fApAbE+k9vj6S0biNze0haoHmQCB9SdpggqthaY4V e9PwuY3FFi9+pddNIlnPaS5VnVEqbJlu4AK385JAfmL1ROdwadNfwEVQiPaAqBOdRMwgXxl z6DcxDo9JzP8h96jLPnYVz3pVU3iTiqAsAfRzBeNpMbIOmVvtUV44U/HWFmz+ScxAGyHFJO Yziq7TKY60dN05FdhPMHw== UI-OutboundReport: notjunk:1;M01:P0:Ij9L9co3VNw=;MHYoAEp/g34ym8TIe7YbH80vFJs yYvwtdpctpW6J2CrNwwFhCVMzmzAAaZR/WG8OByKDUdUsGV64Rul1tdTultGOWeHv9lj+LsJQ 30vTSKGr0pF2hWb0OQirntzlpEbGcgnDlZWOOMOnteFeWhl00M1H1ub5JOfGW385CrTqxGgIf aqaxB2D2l2u5QBOXx7RdUFi92tean10A3Dvh7PZJUiLMWv1lyMQ2gl9UKZ85a3lbTOkn3KG83 lFR5mZycvlyArJLddPfeZO/LtGrCcZTut0QCOj0DpHApgS6OQ0Y61Q6QCWzDNk/Pne3epgSD5 358iicMxKonC7c1y56QGuxHll222e37FQVhL+zpWDq4irwmrVvgb2PJmiA2oVvoPYMv5PW+yj N5HipFeJKtuKZ1l2jSUbDkO9KXypMCmqKWDJ3Dl2gNEdx4j0NQAhGpFNyrUVi8oxx5pqLz1mB rBewS0xxSosWkBsiyF+AtxX2xsaCV6PMIa33xLmlkyC8aTl7gMpwOpwjR8FomxIfFQSggaQt/ AiWHQ2tXQ0ZI7xSo8HqQpi2s8uXmdSqYDAMWJtyOWy0tzdBWffaaGubVPJeIY1jZtpzEgW4ul 7Hpx8ye4b2sOeB1dOnuTEw3RC/VJOZNV0NnNHN/SJBkxKDMIWlBdy5CrYqKlFWI55FXe00hbT J1dJT3XxoZWmshfkhMGPWwNGmkvtJRxWAERf11NLClfqqcuA9kjo7Mz7hbob1MI+hHP9yaQcP mjHPset+iSlTv5pqk8kzZ81gMUMbBEcQnhHR2oevCwkdDgtoI03wP4Ire7MvrFaPK3JsHV8nD GeB1PN3s7mYDT+YkEURfsWLFvBSapRaWqVMZEZn/o0qnps+h6PA/hbBzUDsPNOAL490MLOJpM khru4CU6Sd6UbqeoaUf9J8bE9Q/wgs8jRAuXfT246vqgDo9S/kRTfKCIn6rYTNwl4pvaJ/L97 G+RfWQ== Add test_object_size and its helpers test_loose_object_size and test_packed_object_size, which allow determining the size of a Git object using only the low-level Git commands rev-parse and show-index. Use it in t6300 to replace the bare-bones function test_object_file_size as a motivating example. There it provides the expected output of the high-level Git command for-each-ref. Suggested-by: Jeff King Signed-off-by: René Scharfe --- So how about this? I'm a bit nervous about all the rules about output descriptors and error propagation and whatnot in the test library, but this implementation seems simple enough and might be useful in more than one test. No idea how to add support for alternate object directories, but I doubt we'll ever need it. --- t/t6300-for-each-ref.sh | 16 ++++++-------- t/test-lib-functions.sh | 47 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+), 9 deletions(-) -- 2.43.0 diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh index 843a7fe143..4687660f38 100755 --- a/t/t6300-for-each-ref.sh +++ b/t/t6300-for-each-ref.sh @@ -20,12 +20,6 @@ setdate_and_increment () { export GIT_COMMITTER_DATE GIT_AUTHOR_DATE } -test_object_file_size () { - oid=$(git rev-parse "$1") - path=".git/objects/$(test_oid_to_path $oid)" - test_file_size "$path" -} - test_expect_success setup ' # setup .mailmap cat >.mailmap <<-EOF && @@ -40,7 +34,11 @@ test_expect_success setup ' git branch -M main && setdate_and_increment && git tag -a -m "Tagging at $datestamp" testtag && + testtag_oid=$(git rev-parse refs/tags/testtag) && + testtag_disksize=$(test_object_size $testtag_oid) && git update-ref refs/remotes/origin/main main && + commit_oid=$(git rev-parse refs/heads/main) && + commit_disksize=$(test_object_size $commit_oid) && git remote add origin nowhere && git config branch.main.remote origin && git config branch.main.merge refs/heads/main && @@ -129,7 +127,7 @@ test_atom head push:strip=1 remotes/myfork/main test_atom head push:strip=-1 main test_atom head objecttype commit test_atom head objectsize $((131 + hexlen)) -test_atom head objectsize:disk $(test_object_file_size refs/heads/main) +test_atom head objectsize:disk $commit_disksize test_atom head deltabase $ZERO_OID test_atom head objectname $(git rev-parse refs/heads/main) test_atom head objectname:short $(git rev-parse --short refs/heads/main) @@ -203,8 +201,8 @@ test_atom tag upstream '' test_atom tag push '' test_atom tag objecttype tag test_atom tag objectsize $((114 + hexlen)) -test_atom tag objectsize:disk $(test_object_file_size refs/tags/testtag) -test_atom tag '*objectsize:disk' $(test_object_file_size refs/heads/main) +test_atom tag objectsize:disk $testtag_disksize +test_atom tag '*objectsize:disk' $commit_disksize test_atom tag deltabase $ZERO_OID test_atom tag '*deltabase' $ZERO_OID test_atom tag objectname $(git rev-parse refs/tags/testtag) diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh index 9c3cf12b26..9b49645f77 100644 --- a/t/test-lib-functions.sh +++ b/t/test-lib-functions.sh @@ -1733,6 +1733,53 @@ test_oid_to_path () { echo "${1%$basename}/$basename" } +test_loose_object_size () { + test "$#" -ne 1 && BUG "1 param" + local path=$(test_oid_to_path "$1") + test_file_size "$(git rev-parse --git-path "objects/$path")" 2>&4 +} + +test_packed_object_size () { + test "$#" -ne 2 && BUG "2 params" + local oid=$1 idx=$2 packsize rawsz end + + packsize=$(test_file_size "${idx%.idx}.pack") + rawsz=$(test_oid rawsz) + end=$(($packsize - $rawsz)) + + git show-index <"$idx" | + awk -v oid="$oid" -v end="$end" ' + $2 == oid {start = $1} + {offsets[$1] = 1} + END { + if (!start || start >= end) + exit 1 + for (o in offsets) + if (start < o && o < end) + end = o + print end - start + } + ' && return 0 + + echo >&4 "error: '$oid' not found in '$idx'" + return 1 +} + +test_object_size () { + test "$#" -ne 1 && BUG "1 param" + local oid=$1 + + test_loose_object_size "$oid" 4>/dev/null && return 0 + + for idx in "$(git rev-parse --git-path objects/pack)"/pack-*.idx + do + test_packed_object_size "$oid" "$idx" 4>/dev/null && return 0 + done + + echo >&4 "error: '$oid' not found" + return 1 +} + # Parse oids from git ls-files --staged output test_parse_ls_files_stage_oids () { awk '{print $2}' -