From patchwork Mon Sep 23 08:33:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Passaro via GitGitGadget X-Patchwork-Id: 11156275 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DA896912 for ; Mon, 23 Sep 2019 08:33:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B8C8A2087C for ; Mon, 23 Sep 2019 08:33:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dNts3gIe" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2392596AbfIWId0 (ORCPT ); Mon, 23 Sep 2019 04:33:26 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:43276 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390680AbfIWIdZ (ORCPT ); Mon, 23 Sep 2019 04:33:25 -0400 Received: by mail-wr1-f68.google.com with SMTP id q17so12866560wrx.10 for ; Mon, 23 Sep 2019 01:33:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cqbjFdas5iAFxs9Qje7GsOGHGSNrZCQXW8lY5D5fQg4=; b=dNts3gIe30XlXyl4Prkf/kCvz9jGhUHhRpE4t+36K++VHWqOfYVRfGWKYaav8u+v7F BWrWQAystPtGFRhwxnpzS353KiJkL/ZVQCVmQhy0R9doNXyT2QPpUvirj7TXNC6ZzB/C skaJizYYzuhVtAcSjJ7lcfVm2WnHKrhw19ZDU8inT7O8b71xYdhkT3uEm15lIhdb0dzm spZg9AAIoWIUaIgeptt6I36p76dwQh+CRqxdfqGa+HBtAmKEZvP17Jqolm3jp90sHD59 /Go1sGG8dWIVSPaUqkiza1Ml0QungHB+RV/xPqjB+Nm5u+Raelhc0DajSllHeE+aCGPN n03w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cqbjFdas5iAFxs9Qje7GsOGHGSNrZCQXW8lY5D5fQg4=; b=ti/YMjIZyNSgdls6qMJDuZo27RuC4cP9ZPMJXtno/cbJb69ISvdfEarVEN/vzpPqZ5 sys2En95XzenN5GF4u8MEedDVm9iYNwMCt0SWe/3c4/xW3Q3XmRC9+TVGl7zQE/BKIWZ dXXmHVJZBkKSlvHW89TenBQ4xb6qt4JAmLjR2BGqoASpd6LEwzIzlrFTPZE6l8/zjbB1 ugI89TW5IWB6Q/6ByvH/W2CgJqJDfxoPbS9Vas/sA8Zm++ho+D9Q8j47YNoezcbPGr7b mDhRibBfaVL3jZyLNDCT++QWd9eJZVqJdh/XRkNIVyBtvCrRXWSrKS4yIsLKYpK3XBup 5hpA== X-Gm-Message-State: APjAAAVZw8EcY2J+g76RiBGB2UBnobIrbUPg4GA1Jv+Uwfhr7U/osMtQ 5v2PdZ+NA6EU45HXDCy9I9qoBcic X-Google-Smtp-Source: APXvYqxqgJS7tfktj8LZ0z0urUoVscsVbxY1rOYZycQBnfVk7yw2wAqktdRDJmQar1mVasGRS4CyrA== X-Received: by 2002:a5d:6302:: with SMTP id i2mr9549926wru.249.1569227603325; Mon, 23 Sep 2019 01:33:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d9sm13508110wrf.62.2019.09.23.01.33.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 23 Sep 2019 01:33:22 -0700 (PDT) Date: Mon, 23 Sep 2019 01:33:22 -0700 (PDT) X-Google-Original-Date: Mon, 23 Sep 2019 08:33:20 GMT Message-Id: In-Reply-To: References: From: "Alexandr Miloslavskiy via GitGitGadget" Subject: [PATCH 1/2] t0028: fix test for UTF-16-LE-BOM Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Alexandr Miloslavskiy Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Alexandr Miloslavskiy According to its name, the test its designed for UTF-16-LE-BOM. However, possibly due to copy&paste oversight, it was using UTF-32 file. While the test succeeds (probably interprets extra \x00\x00 as embedded zero), I myself had an unrelated problem which caused the test to fail. When analyzing the failure I was quite puzzled by the fact that the test is obviously bugged. And it seems that I'm not alone: https://public-inbox.org/git/CAH8yC8kSakS807d4jc_BtcUJOrcVT4No37AXSz=jePxhw-o9Dg@mail.gmail.com/T/#u This fix changes the test to follow its original intention. Signed-off-by: Alexandr Miloslavskiy --- t/t0028-working-tree-encoding.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t0028-working-tree-encoding.sh b/t/t0028-working-tree-encoding.sh index 1090e650ed..5493cf3ca9 100755 --- a/t/t0028-working-tree-encoding.sh +++ b/t/t0028-working-tree-encoding.sh @@ -40,7 +40,7 @@ test_expect_success 'setup test files' ' printf "$text" | write_utf16 >test.utf16.raw && printf "$text" | write_utf32 >test.utf32.raw && printf "\377\376" >test.utf16lebom.raw && - printf "$text" | iconv -f UTF-8 -t UTF-32LE >>test.utf16lebom.raw && + printf "$text" | iconv -f UTF-8 -t UTF-16LE >>test.utf16lebom.raw && # Line ending tests printf "one\ntwo\nthree\n" >lf.utf8.raw && From patchwork Mon Sep 23 08:33:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Passaro via GitGitGadget X-Patchwork-Id: 11156277 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 96815112B for ; Mon, 23 Sep 2019 08:33:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7477B2064A for ; Mon, 23 Sep 2019 08:33:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZUKD6ZtL" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2404845AbfIWId1 (ORCPT ); Mon, 23 Sep 2019 04:33:27 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:33164 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391785AbfIWId0 (ORCPT ); Mon, 23 Sep 2019 04:33:26 -0400 Received: by mail-wm1-f65.google.com with SMTP id r17so14416359wme.0 for ; Mon, 23 Sep 2019 01:33:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=stbWo9RCfPeujJ9kOQ2CfIq+snWvntoSqePPwb6tozo=; b=ZUKD6ZtLimrzgmN/FJHXcvZUBKrm23A5/lEsMPV7KUYPKVFkfNzcTX28iVGFcnbZ9d UG0zn9Jo5KW6C9CLFGa2GDI2fT9Px1w4XC6915MuTarFljENzowkeKuU8RJuSzG9JGFU EWApaEZ5iQSSf4ZGULaWEjehcOeSaw9g1I4bqOo85S5vvUM8hQC0Yzi7KBkz60ynLmh6 7SPpaey+KlNeDSO9ZZc2rM4+IMehw9WjzmeOHjyTL71WvdKDbsxO3NLM1PNpKEykprH/ +r+2SFDDyi0+pjqOkAKeccI7WVFnxrnMfZ+GHk+U4TI4zmF8rtq3pduOXNMD6dWZy0U6 UoZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=stbWo9RCfPeujJ9kOQ2CfIq+snWvntoSqePPwb6tozo=; b=lICoauCN+q/XAw9Q9+I7ozWj5MvUTr7/asVtXBYmq9g7F3a9AD1qcNwFZ/ZH4iW01P LUf7wV53kWL2zjSGNkHp4EyclQonAnpCkZXwdCZYrkjrdXqfFSSHHTbeaL+n1qKYWbiV ATt5X/P1JNFrOxh7mOfeA51/yCSa298xBxl+6CGKqPN7cDSgX3YtXbqJun5l77JUZ0RG ++HbwaRRKfqgi4iZmBQd+10xUFStYORrLe/jOWA0qAJnovnI8sLJG8cDOUCSEh4nlx29 nTmbQ2uTiAfRuM8nykk6ZPxXsOVdWiQgtqZjg/Q3krKr9qE03MVwgQbebRs78Wah7YnO 10RQ== X-Gm-Message-State: APjAAAWP2jfpfEQlkU7gTsnINgaHJiJ5XW1Lapb17zu95jQilWk3wlSm vJQvlGSHCPLhSy77xmMD20eZvrI/ X-Google-Smtp-Source: APXvYqwHJtK/vFC3ewkmBFiogMAP3TQYnOp6fdSKswLsKEfL3C7zjlNXPjZDUdw88UiuXlra228hWQ== X-Received: by 2002:a1c:a942:: with SMTP id s63mr12782440wme.152.1569227603957; Mon, 23 Sep 2019 01:33:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g73sm16214617wme.10.2019.09.23.01.33.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 23 Sep 2019 01:33:23 -0700 (PDT) Date: Mon, 23 Sep 2019 01:33:23 -0700 (PDT) X-Google-Original-Date: Mon, 23 Sep 2019 08:33:21 GMT Message-Id: In-Reply-To: References: From: "Alexandr Miloslavskiy via GitGitGadget" Subject: [PATCH 2/2] t0028: add more tests Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Alexandr Miloslavskiy Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Alexandr Miloslavskiy After I discovered that UTF-16-LE-BOM test was bugged and still succeeded, I decided that better tests are required. Possibly the best option here is to compare git results against hardcoded ground truth. The new tests also cover more interesting chars where (ANSI != UTF-8). Signed-off-by: Alexandr Miloslavskiy --- t/t0028-working-tree-encoding.sh | 39 ++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/t/t0028-working-tree-encoding.sh b/t/t0028-working-tree-encoding.sh index 5493cf3ca9..d0dd5dd0ea 100755 --- a/t/t0028-working-tree-encoding.sh +++ b/t/t0028-working-tree-encoding.sh @@ -280,4 +280,43 @@ test_expect_success ICONV_SHIFT_JIS 'check roundtrip encoding' ' git reset ' +# $1: checkout encoding +# $2: test string +# $3: binary test string in checkout encoding +test_commit_utf8_checkout_other () { + encoding="$1" + orig_string="$2" + expect_bytes="$3" + + test_expect_success "Commit utf-8, checkout ${encoding}" ' + test_when_finished "git checkout HEAD -- .gitattributes" && + + test_ext="commit_utf8_checkout_${encoding}" && + test_file="test.${test_ext}" && + + # Commit as utf-8 + echo "*.${test_ext} text working-tree-encoding=utf-8" >.gitattributes && + printf "${orig_string}" >"${test_file}" && + git add "${test_file}" && + git commit -m "Test data" && + + # Checkout in tested encoding + rm "${test_file}" && + echo "*.${test_ext} text working-tree-encoding=${encoding}" >.gitattributes && + git checkout HEAD -- "${test_file}" && + + # Test + printf "${expect_bytes}" > "${test_file}.raw" && + test_cmp_bin "${test_file}.raw" "${test_file}" + ' +} + +test_commit_utf8_checkout_other "UTF-8" "Test Тест" "\124\145\163\164\040\320\242\320\265\321\201\321\202" +test_commit_utf8_checkout_other "UTF-16LE" "Test Тест" "\124\000\145\000\163\000\164\000\040\000\042\004\065\004\101\004\102\004" +test_commit_utf8_checkout_other "UTF-16BE" "Test Тест" "\000\124\000\145\000\163\000\164\000\040\004\042\004\065\004\101\004\102" +test_commit_utf8_checkout_other "UTF-16LE-BOM" "Test Тест" "\377\376\124\000\145\000\163\000\164\000\040\000\042\004\065\004\101\004\102\004" +test_commit_utf8_checkout_other "UTF-16BE-BOM" "Test Тест" "\376\377\000\124\000\145\000\163\000\164\000\040\004\042\004\065\004\101\004\102" +test_commit_utf8_checkout_other "UTF-32LE" "Test Тест" "\124\000\000\000\145\000\000\000\163\000\000\000\164\000\000\000\040\000\000\000\042\004\000\000\065\004\000\000\101\004\000\000\102\004\000\000" +test_commit_utf8_checkout_other "UTF-32BE" "Test Тест" "\000\000\000\124\000\000\000\145\000\000\000\163\000\000\000\164\000\000\000\040\000\000\004\042\000\000\004\065\000\000\004\101\000\000\004\102" + test_done