From patchwork Sat Dec 7 17:47:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277691 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F2C09159A for ; Sat, 7 Dec 2019 17:47:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D09B924673 for ; Sat, 7 Dec 2019 17:47:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Puff+zLL" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726489AbfLGRrs (ORCPT ); Sat, 7 Dec 2019 12:47:48 -0500 Received: from mail-wr1-f67.google.com ([209.85.221.67]:47067 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726414AbfLGRrs (ORCPT ); Sat, 7 Dec 2019 12:47:48 -0500 Received: by mail-wr1-f67.google.com with SMTP id z7so11228288wrl.13 for ; Sat, 07 Dec 2019 09:47:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=z5bbmaVjWTEBlGwbALe96MP/a+9xy4gA1x7Nw3wmnbY=; b=Puff+zLLNJADWGfkNzX7flizrxMusHzlWv8UksvmTv3oWfFiv0ndvYBWZGqSG90a0M ToKZOFxYyYiUtic5NAexYRMHD5czVHd7myfLjbGP3gTrL5wgUG9V2njaG6StHkFbfIiL GQoGIUZTWP5HwZiL9u9b/vzg0MV6DnsOrLYoGDkp3lbFuS5ml6ZYfBWiguHquFujZDWo 2QP16HpnfZzW4oD6cjLA54O+9s+n78xyuOQ9tKRCJkhhmQgaMuEaAfm9sKVdcBWIaPk5 dCDE0mzv59pOAeJcH6DXm0oF6W8Nth69aTEKtHYXvPGzifL2++Scn3mNbBkhpxQPk9DD 4Dgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=z5bbmaVjWTEBlGwbALe96MP/a+9xy4gA1x7Nw3wmnbY=; b=P/5k1pa3ebMVi73W5zoFSDRUwwQsi9ZJ1EGIWVIDsFeGIJBSeFSc/ThKk2YiASdIAK 0uydDXVIO3Pcp1oNXvC0HqExU6SdEkBDQ2nE/8vnEkh6RdPvY6ewp54NfWIkWYtC0w0h DnBC5VOYGFJXbKZCBKHIpB0VfPKIB2WQiHfzlxiLkRqXW1cgEdfZVbTdr/A4gQTu/WXd sCQtHHjFnDwuxQS6s19+XXXPQvOTOeiFt9c47M8BujwPckzbLhaXxsiQiVeDM1Z+1d4w XFZI2Slt94kZ8ne88W4nm7hArTpCX4KxL1TsLpkN2zlUI/S5GDkRMvxonGjQwgPzw7Nq AvnQ== X-Gm-Message-State: APjAAAVBZuKLW+ftwBJIo1yYgoEF/S/rJ51iyN2h8WqDPvhp15QMtW1g d1tA5Vtp2zuiJzNaFAucPFulFsJ6 X-Google-Smtp-Source: APXvYqzgv70K6eWJTyVIJcYCj04/R3bKVVZBZbtbXnYcRcHNHauIKUMbcLCJi7qbsrDHKSgmMpGhWA== X-Received: by 2002:adf:e887:: with SMTP id d7mr21174827wrm.162.1575740866121; Sat, 07 Dec 2019 09:47:46 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l17sm7485357wme.45.2019.12.07.09.47.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:45 -0800 (PST) Message-Id: In-Reply-To: References: From: "Jeff King via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:29 +0000 Subject: [PATCH v5 01/15] t/gitweb-lib.sh: drop confusing quotes Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Jeff King Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King Some variables assignments in gitweb_run() look like this: FOO=""$1"" The extra quotes aren't doing anything. Each set opens and closes an empty string, and $1 is actually outside of any double-quotes (which is OK, because variable assignment does not do whitespace splitting on the expanded value). Let's drop them, as they're simply confusing. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano --- t/gitweb-lib.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/t/gitweb-lib.sh b/t/gitweb-lib.sh index 1f32ca66ea..b8455d1182 100644 --- a/t/gitweb-lib.sh +++ b/t/gitweb-lib.sh @@ -60,7 +60,10 @@ gitweb_run () { REQUEST_METHOD='GET' QUERY_STRING=$1 PATH_INFO=$2 +<<<<<<< HEAD REQUEST_URI=/gitweb.cgi$PATH_INFO +======= +>>>>>>> t/gitweb-lib.sh: drop confusing quotes export GATEWAY_INTERFACE HTTP_ACCEPT REQUEST_METHOD \ QUERY_STRING PATH_INFO REQUEST_URI From patchwork Sat Dec 7 17:47:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277695 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 53D3F139A for ; Sat, 7 Dec 2019 17:48:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 32982217BA for ; Sat, 7 Dec 2019 17:48:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IS9iNi8Z" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726552AbfLGRru (ORCPT ); Sat, 7 Dec 2019 12:47:50 -0500 Received: from mail-wr1-f66.google.com ([209.85.221.66]:41772 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726469AbfLGRrt (ORCPT ); Sat, 7 Dec 2019 12:47:49 -0500 Received: by mail-wr1-f66.google.com with SMTP id c9so11318093wrw.8 for ; Sat, 07 Dec 2019 09:47:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=02wPiKcxW3MLMy/ukr6QS7zsK/Dkfk9fBmn17gWfJQA=; b=IS9iNi8ZoentsM3ihrcWileJB0C9Lbz7x+aRP2+Q7tQbyx1qQMcLPV6cksJ42MBMd0 vMLMHJUs4QhRQsMwEaK4cQ6Bc5oFXZLMeApk+BgdWJmfoMA99m9i7KpJ04NrbFQD1OwT iTM4NxKaQbDQVW8m0psfpkk6jtWzL92ZwFJ8TRy5+f3piwtjZHsGLdgl3nc7ZqaVVgMX UT2wMt9ATlcNdMMki6UijHxqfn7Ubi0stVL3W3NcjxOBB4PwFUmtz/+CNsWpd6HJNISl sYDNxBLQObAJnsKJF4de4Poq1ayksscBUQYNvL0Fyvhij3Pnp95C6GfjA7fc2oH6zN3A Q3Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=02wPiKcxW3MLMy/ukr6QS7zsK/Dkfk9fBmn17gWfJQA=; b=gVJD3d1zJIjNFSUQC9aijUu50yjiexHHWvpSgCel1/uuwnhbJKdAwy6Sa2QbdYqx7A Y2jmDRcWzchC8QAPUvD6y36s0zdemMQpjDEooVzc20WGst2pA2y+nVmlPIdQ2xegoMlv VfCzN0w6g8YNeTmQ0eys0SJlsUlbwHCCZ2pItrTlSyfc3UW+zVEpn1eVxu6VwRqjsQH3 TSXfGwiRobq4njP9fTWMhKdPu6HxhYg9CsqHMz+TIWiFUz8SQje2VhBoh1uYENZkV2sB QggzdhcqtIrUooJ2NQivvtN97sCawzEJXFb79y2sH0HnmHg9sJjTg7gMsAakbVM4cr8I K0oA== X-Gm-Message-State: APjAAAXyqOSBZhjuAk2zdU9RIZU7sZKX8g8DvXlT0og0oUC6gSTptlUZ 8gJOpwsktFzFdtPY7I5cMdDY6M25 X-Google-Smtp-Source: APXvYqyrOEOeGrlQ2cicyfI0Gq/INE6+drn9OCt8+Eve25wiPlQs1EeNTgpNP5H9J5gcoW1Z68LoYg== X-Received: by 2002:a05:6000:1241:: with SMTP id j1mr23210063wrx.26.1575740867415; Sat, 07 Dec 2019 09:47:47 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w22sm7181937wmk.34.2019.12.07.09.47.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:47 -0800 (PST) Message-Id: In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:31 +0000 Subject: [PATCH v5 03/15] git-p4: select P4 binary by operating-system Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene The original code unconditionally used "p4" as the binary filename. Depending on the version of Git and Python installed, the perforce program (p4) may not resolve on Windows without the program extension. Check the operating system (platform.system) and if it is reporting that it is Windows, use the full filename of "p4.exe" instead of "p4" This change is Python 2 and Python 3 compatible. Signed-off-by: Ben Keene --- git-p4.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/git-p4.py b/git-p4.py index 60c73b6a37..65e926758c 100755 --- a/git-p4.py +++ b/git-p4.py @@ -75,7 +75,10 @@ def p4_build_cmd(cmd): location. It means that hooking into the environment, or other configuration can be done more easily. """ - real_cmd = ["p4"] + if (platform.system() == "Windows"): + real_cmd = ["p4.exe"] + else: + real_cmd = ["p4"] user = gitConfig("git-p4.user") if len(user) > 0: From patchwork Sat Dec 7 17:47:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277697 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7EC30186D for ; Sat, 7 Dec 2019 17:48:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5D4FD217BA for ; Sat, 7 Dec 2019 17:48:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cUdPhxE9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726578AbfLGRrv (ORCPT ); Sat, 7 Dec 2019 12:47:51 -0500 Received: from mail-wr1-f46.google.com ([209.85.221.46]:46758 "EHLO mail-wr1-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726414AbfLGRrv (ORCPT ); Sat, 7 Dec 2019 12:47:51 -0500 Received: by mail-wr1-f46.google.com with SMTP id z7so11228348wrl.13 for ; Sat, 07 Dec 2019 09:47:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0dqqa6jkHEt5zsxaZPmqLAX5RIVwpEdWH2EzXL8V48Q=; b=cUdPhxE9owG7EVmSlR5RMNqBqBmR1+1xBNC0ACbBJcfdK4mx6dSYF34uBxrqfTlxEX iuClDiiXXl/OqE/FP3oQwNhHq1J/pAsNMx+iyaHC4GNhyZZFuSh2v1NI3xswpeUoO5RV gF2HWauKBBGkbm0NL6SFzhooT913jHmRK121aGy10cGbNZ14VT5PWLqNNRjwu0T0mAsG vKwU6+FCLS9r/Zmuff3K3T9xVZvGtdXFxrgnzfYJ5ivn2TFXfvmXOD7VWVpkj5+NuP9v Q+GVxaIzeFz5NoHja/0TlcWFhKxm+SQIXyoehPxOGdZjkPWAhEqJ6gN8tR4VxUURxQji zSrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0dqqa6jkHEt5zsxaZPmqLAX5RIVwpEdWH2EzXL8V48Q=; b=caeNWwOW2CLeDTB+CtUTpJLID/xMZobcRHlo4Em5TpfrEkhpVMWXXM4O7JeXyJ2G69 Z3YL61lVOFgTOmrOURalYykJACncaeHzdnlnkwKqN4RcjCvwZsJDW00oop47q0VenOcV lGyG1Xi/JsAwnhj3RP3MzX0ld4r3nCkS5va1Pt080FYFpVK2QAHq+2dtw9l/PWAicaEQ DxGFH9i9MIYraruKjycINNPteI7Jv4zCNN9J8cyl+SzfvYv724sKLvb7F+BfHQnOHHpr rBTRjrl6KMsEIiWn6XrsK8mML8qoK79c0iH6q2C276fdRDMQqKY98iCUV0JQuqnfh0sg R4og== X-Gm-Message-State: APjAAAX2nuXbvmBKSw1/au+4e82i/uoz7SU3hUTFsDKKVgWLfrTeZxPn 6tdRwMzDwEFOUAngvpP+Nh+kVnLt X-Google-Smtp-Source: APXvYqxzzu8kj9XglDJSAtFpYndb15GzalIYD6NZ4wJ3BuLerrKPXGdTbw7nFkW7GfclTGHWiD8/8g== X-Received: by 2002:adf:d184:: with SMTP id v4mr20716388wrc.76.1575740868227; Sat, 07 Dec 2019 09:47:48 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s15sm17759449wrp.4.2019.12.07.09.47.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:47 -0800 (PST) Message-Id: <7170aface2270e8c46439c5c1e01d2b18cdf6fd0.1575740863.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:32 +0000 Subject: [PATCH v5 04/15] git-p4: change the expansion test from basestring to list Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene Python 3 handles strings differently than Python 2.7. Since Python 2 is reaching it's end of life, a series of changes are being submitted to enable python 3.5 and following support. The current code fails basic tests under python 3.5. The original code used 'basestring' in a test to determine if a list or literal string was passed into 9 different functions. This is used to determine if the shell should be invoked when calling subprocess methods. Change references to 'basestring' in the isinstance tests to use 'list' instead. This prepares the code to remove all references to basestring. Signed-off-by: Ben Keene --- git-p4.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/git-p4.py b/git-p4.py index 65e926758c..3153186df0 100755 --- a/git-p4.py +++ b/git-p4.py @@ -108,7 +108,7 @@ def p4_build_cmd(cmd): # Provide a way to not pass this option by setting git-p4.retries to 0 real_cmd += ["-r", str(retries)] - if isinstance(cmd,basestring): + if not isinstance(cmd, list): real_cmd = ' '.join(real_cmd) + ' ' + cmd else: real_cmd += cmd @@ -174,7 +174,7 @@ def write_pipe(c, stdin): if verbose: sys.stderr.write('Writing pipe: %s\n' % str(c)) - expand = isinstance(c,basestring) + expand = not isinstance(c, list) p = subprocess.Popen(c, stdin=subprocess.PIPE, shell=expand) pipe = p.stdin val = pipe.write(stdin) @@ -196,7 +196,7 @@ def read_pipe_full(c): if verbose: sys.stderr.write('Reading pipe: %s\n' % str(c)) - expand = isinstance(c,basestring) + expand = not isinstance(c, list) p = subprocess.Popen(c, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=expand) (out, err) = p.communicate() return (p.returncode, out, err) @@ -232,7 +232,7 @@ def read_pipe_lines(c): if verbose: sys.stderr.write('Reading pipe: %s\n' % str(c)) - expand = isinstance(c, basestring) + expand = not isinstance(c, list) p = subprocess.Popen(c, stdout=subprocess.PIPE, shell=expand) pipe = p.stdout val = pipe.readlines() @@ -275,7 +275,7 @@ def p4_has_move_command(): return True def system(cmd, ignore_error=False): - expand = isinstance(cmd,basestring) + expand = not isinstance(cmd, list) if verbose: sys.stderr.write("executing %s\n" % str(cmd)) retcode = subprocess.call(cmd, shell=expand) @@ -287,7 +287,7 @@ def system(cmd, ignore_error=False): def p4_system(cmd): """Specifically invoke p4 as the system command. """ real_cmd = p4_build_cmd(cmd) - expand = isinstance(real_cmd, basestring) + expand = not isinstance(real_cmd, list) retcode = subprocess.call(real_cmd, shell=expand) if retcode: raise CalledProcessError(retcode, real_cmd) @@ -525,7 +525,7 @@ def getP4OpenedType(file): # Return the set of all p4 labels def getP4Labels(depotPaths): labels = set() - if isinstance(depotPaths,basestring): + if not isinstance(depotPaths, list): depotPaths = [depotPaths] for l in p4CmdList(["labels"] + ["%s..." % p for p in depotPaths]): @@ -612,7 +612,7 @@ def isModeExecChanged(src_mode, dst_mode): def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False, errors_as_exceptions=False): - if isinstance(cmd,basestring): + if not isinstance(cmd, list): cmd = "-G " + cmd expand = True else: @@ -629,7 +629,7 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False, stdin_file = None if stdin is not None: stdin_file = tempfile.TemporaryFile(prefix='p4-stdin', mode=stdin_mode) - if isinstance(stdin,basestring): + if not isinstance(stdin, list): stdin_file.write(stdin) else: for i in stdin: From patchwork Sat Dec 7 17:47:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277699 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F2CB81892 for ; Sat, 7 Dec 2019 17:48:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D1A50217BA for ; Sat, 7 Dec 2019 17:48:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ELNjeZMZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726626AbfLGRry (ORCPT ); Sat, 7 Dec 2019 12:47:54 -0500 Received: from mail-wr1-f43.google.com ([209.85.221.43]:43869 "EHLO mail-wr1-f43.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726528AbfLGRrw (ORCPT ); Sat, 7 Dec 2019 12:47:52 -0500 Received: by mail-wr1-f43.google.com with SMTP id d16so11276526wre.10 for ; Sat, 07 Dec 2019 09:47:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=30grMjwfxMpq58Y+pNq9Q1EJrfZE2e38Cnol6jyt55s=; b=ELNjeZMZZiuRWNWM2Q0n838OsNCtXsUIRbTQl22hcaIppZVd46PCby4fGtUrFVNK16 9Ro/zZj5WC5FDZxjDaOs74eNRtBvYyfSe0i5G5EAMw5NMdd3jEYSeHclW/OLmPP+eqJv 426ixmr/MK6evPhSLPomVeMLIk+y9zxucMsL0d9XxKKdKByTdZfKT5AeJN4WdTGcX+lI FBFPx/G0zCyhYp8Mo/UtqyHFHCa2X18bdwee3oim7+7EA3w5iRxz/Hgn1GNr2zIF982r +1cMwyCHfoQvg+iDMs1HEd4d0gvQtnwIeDDZThQFdzblWWZwkmFCLYC0xtZ2FbWy/DiF Px/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=30grMjwfxMpq58Y+pNq9Q1EJrfZE2e38Cnol6jyt55s=; b=YKd9kR8H6VTOMr96zg22WYTW6HrkNMkZ78gUVXvYerMduExX5P+xKytJwJ9DlDRV8g qH8VvJJsoIkttoL0llPO/S1xuK+R/SNxOSTAnqTH5tKfW8AIQtLOxDZ19Clsby6OeXpq 8tD9AtQSQq9sZHcn5KYKbY1BGy8I8JaV5C9woGsfuLUeXkbnDx77mRDRaf8q/wMWKq4e tbjAjhwEGcFW615MMigabjSTBGZPPCJsPtNJRYjmDdbtnNbJdOQES/yVzq+qEB0oPNXP MdAWNjJ14EB1N5Y6nZ0f1NgTFQ2FzBBolcovfyZNWvh3NBcne9LhxH81PaaBuTAL9xf+ /J3w== X-Gm-Message-State: APjAAAWhCcBAf2CFhgyCWFduxGnAbq69ZDUgolK+ieWFQ9fIl92EXjDf kbExmA9Y9mnIbAiF3spa9WaIPKkJ X-Google-Smtp-Source: APXvYqx+9leW9o2Hk8ZnVcMP5nDH7oA37ChRsoks3mkI240R1NJJpHnVKnTdtBSNidFkq8yYXT2Isg== X-Received: by 2002:adf:ed83:: with SMTP id c3mr19501254wro.51.1575740868908; Sat, 07 Dec 2019 09:47:48 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a5sm1757918wmb.37.2019.12.07.09.47.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:48 -0800 (PST) Message-Id: <11d7703e411f1dced8a34defc68922ba44c614d5.1575740863.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:33 +0000 Subject: [PATCH v5 05/15] git-p4: promote encodeWithUTF8() to a global function Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene This changelist is an intermediate submission for migrating the P4 support from Python 2 to Python 3. The code needs access to the encodeWithUTF8() for support of non-UTF8 filenames in the clone class as well as the sync class. Move the function encodeWithUTF8() from the P4Sync class to a stand-alone function. This will allow other classes to use this function without instanciating the P4Sync class. Change the self.verbose reference to an optional method parameter. Update the existing references to this function to pass the self.verbose since it is no longer available on "self" since the function is no longer contained on the P4Sync class. Signed-off-by: Ben Keene --- git-p4.py | 52 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 36 insertions(+), 16 deletions(-) diff --git a/git-p4.py b/git-p4.py index 3153186df0..cc6c490e2c 100755 --- a/git-p4.py +++ b/git-p4.py @@ -27,7 +27,7 @@ import ctypes import errno -# support basestring in python3 +# support basestring in Python 3 try: unicode = unicode except NameError: @@ -46,7 +46,7 @@ try: from subprocess import CalledProcessError except ImportError: - # from python2.7:subprocess.py + # from Python 2.7:subprocess.py # Exception classes used by this module. class CalledProcessError(Exception): """This exception is raised when a process run by check_call() returns @@ -587,6 +587,38 @@ def isModeExec(mode): # otherwise False. return mode[-3:] == "755" +def encodeWithUTF8(path, verbose=False): + """ Ensure that the path is encoded as a UTF-8 string + + Returns bytes(P3)/str(P2) + """ + + if isunicode: + try: + if isinstance(path, unicode): + # It is already unicode, cast it as a bytes + # that is encoded as utf-8. + return path.encode('utf-8', 'strict') + path.decode('ascii', 'strict') + except: + encoding = 'utf8' + if gitConfig('git-p4.pathEncoding'): + encoding = gitConfig('git-p4.pathEncoding') + path = path.decode(encoding, 'replace').encode('utf8', 'replace') + if verbose: + print('\nNOTE:Path with non-ASCII characters detected. Used %s to encode: %s ' % (encoding, to_unicode(path))) + else: + try: + path.decode('ascii') + except: + encoding = 'utf8' + if gitConfig('git-p4.pathEncoding'): + encoding = gitConfig('git-p4.pathEncoding') + path = path.decode(encoding, 'replace').encode('utf8', 'replace') + if verbose: + print('Path with non-ASCII characters detected. Used %s to encode: %s ' % (encoding, path)) + return path + class P4Exception(Exception): """ Base class for exceptions from the p4 client """ def __init__(self, exit_code): @@ -2748,24 +2780,12 @@ def writeToGitStream(self, gitMode, relPath, contents): self.gitStream.write(d) self.gitStream.write('\n') - def encodeWithUTF8(self, path): - try: - path.decode('ascii') - except: - encoding = 'utf8' - if gitConfig('git-p4.pathEncoding'): - encoding = gitConfig('git-p4.pathEncoding') - path = path.decode(encoding, 'replace').encode('utf8', 'replace') - if self.verbose: - print('Path with non-ASCII characters detected. Used %s to encode: %s ' % (encoding, path)) - return path - # output one file from the P4 stream # - helper for streamP4Files def streamOneP4File(self, file, contents): relPath = self.stripRepoPath(file['depotFile'], self.branchPrefixes) - relPath = self.encodeWithUTF8(relPath) + relPath = encodeWithUTF8(relPath, self.verbose) if verbose: if 'fileSize' in self.stream_file: size = int(self.stream_file['fileSize']) @@ -2848,7 +2868,7 @@ def streamOneP4File(self, file, contents): def streamOneP4Deletion(self, file): relPath = self.stripRepoPath(file['path'], self.branchPrefixes) - relPath = self.encodeWithUTF8(relPath) + relPath = encodeWithUTF8(relPath, self.verbose) if verbose: sys.stdout.write("delete %s\n" % relPath) sys.stdout.flush() From patchwork Sat Dec 7 17:47:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277701 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2BD21186E for ; Sat, 7 Dec 2019 17:48:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0818B217BA for ; Sat, 7 Dec 2019 17:48:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VrHg9MaW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726635AbfLGRrz (ORCPT ); Sat, 7 Dec 2019 12:47:55 -0500 Received: from mail-wm1-f54.google.com ([209.85.128.54]:34714 "EHLO mail-wm1-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726469AbfLGRrw (ORCPT ); Sat, 7 Dec 2019 12:47:52 -0500 Received: by mail-wm1-f54.google.com with SMTP id f4so12115203wmj.1 for ; Sat, 07 Dec 2019 09:47:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=X9OallQF0hIpgkeVK60DD1vy4QlaqOZC2W/XfzNZN08=; b=VrHg9MaWzsCvCInBk5x6XsQWKhsR5M8Lo44DZp71dKxfYW4bAOdPvZ0l9KyXxUK+Mf IkHZ4kXKejOmdcCYPaHaHlY0n7YhTUmOeu+0Ej/l/2fRGK7bel43IqMGVPTTDi6H3CSA bK6AxQVdPB4LhKI5cyBR6ZHdLtr/JCDFtfg0q3M4bSl2ErtN1hFXDx4s0XJqCt9r/icy eKSW8xTz5Pe42uD/lKb/ismJq+MmLSq4UtrgKSc8UA4fbYBgMWVRKdOT0zLMgi1HuNVY HwSC99ryVNZGc8QEz0+YsGA8lgQ4Si9ca34fl0hkrIH2eyaDB17I6ZbmrqFQ4zyzQ7Vs Izsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=X9OallQF0hIpgkeVK60DD1vy4QlaqOZC2W/XfzNZN08=; b=pjvOR/lXudNTtAwO6MpBb3Gu2cCwZSNTC6vWovDBxTHLoahU7Ra4VOi1aVvnO2v7z5 iEr15WUCu8/FS0KRtvP7xcQoOQzTZVRsHJpO2VM9PpRnyk5eDoBzvVlMsz/9aQ/Nnrkt k85gE3epe8uEaldw02NhM74MS6r+3DUfvOX7kmaGEiIkMfqTkyMfhBkmceDFsRUaaUAG qnYGKRgt/dMSG9mobuTUcBXMV0WIKw4YNDbcrPL68Ma0ioAHymKkWSRAAgP0dpuBatyu do+rhMNi6ru7MrEdGwsjLVS6JVqtm6QCxdJllQfkDzNDb3ttBSThnazs9gAojwbKnla1 lqyg== X-Gm-Message-State: APjAAAV7Dn0thwg9UxncYLTU5dyyP4jCJvHBCGSfaeJXpogMIFQw2gcT tyecUlsSoSoXInuTepHbwjJEKdEU X-Google-Smtp-Source: APXvYqw8SMO43YdtbSe+HXNoXVyl4h0dnVfKtTRP1LwaDwxdakcBzAeoynJz1MEmgiqOdJHXXgCVgw== X-Received: by 2002:a7b:c85a:: with SMTP id c26mr16606850wml.107.1575740869639; Sat, 07 Dec 2019 09:47:49 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f12sm7098876wmf.28.2019.12.07.09.47.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:49 -0800 (PST) Message-Id: In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:34 +0000 Subject: [PATCH v5 06/15] git-p4: remove p4_write_pipe() and write_pipe() return values Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene The git-p4 functions write_pipe() and p4_write_pipe() originally return the number of bytes returned from the system call. However, this is a misleading value when this function is run by Python 3. Modify the functions write_pipe() and p4_write_pipe() to remove the return value. The return value for both functions is the number of bytes, but the meaning is lost under python3 since the count does not match the number of characters that may have been encoded. Additionally, the return value was never used, so this is removed to avoid future ambiguity. Signed-off-by: Ben Keene --- git-p4.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/git-p4.py b/git-p4.py index cc6c490e2c..e7c24817ad 100755 --- a/git-p4.py +++ b/git-p4.py @@ -171,6 +171,8 @@ def die(msg): sys.exit(1) def write_pipe(c, stdin): + """ Executes the command 'c', passing 'stdin' on the standard input + """ if verbose: sys.stderr.write('Writing pipe: %s\n' % str(c)) @@ -182,11 +184,12 @@ def write_pipe(c, stdin): if p.wait(): die('Command failed: %s' % str(c)) - return val def p4_write_pipe(c, stdin): + """ Runs a P4 command 'c', passing 'stdin' data to P4 + """ real_cmd = p4_build_cmd(c) - return write_pipe(real_cmd, stdin) + write_pipe(real_cmd, stdin) def read_pipe_full(c): """ Read output from command. Returns a tuple From patchwork Sat Dec 7 17:47:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277717 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5AB7B139A for ; Sat, 7 Dec 2019 17:48:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3634C24673 for ; Sat, 7 Dec 2019 17:48:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="idW8kX2y" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726608AbfLGRrx (ORCPT ); Sat, 7 Dec 2019 12:47:53 -0500 Received: from mail-wr1-f66.google.com ([209.85.221.66]:47073 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726555AbfLGRrw (ORCPT ); Sat, 7 Dec 2019 12:47:52 -0500 Received: by mail-wr1-f66.google.com with SMTP id z7so11228403wrl.13 for ; Sat, 07 Dec 2019 09:47:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=czCk0DRp2OOPFajQ4NWb1LoJtE/xpLksVA/HZQ+botI=; b=idW8kX2yOQZBKQxyNbXBvUUHEq4+IIQCvOlGTDjvB7pBNRzMd9gceldcFUYcUA2Hht 4PejE9P6vga4FTmrBEVMarzFlsXf8Uc8xZQzXfV0iI0xiMYTgGshEiJ8atR+kCaLLvxp 8Ta/P7tPQRMluw3L4A0cSnfQfkd5wS/EFxBmpy5JIKw+ebNCUz9ewkLPrewWTZk5IWgN 18IaboNbV6mq3JNe27BaHJA06CWdWEbOZP3A932WO2T9EJ4o+cBGLxSPCoki1aawIqHq MCBRy2rbQT8aJatbJOzVCEJKfoKKUzLrL6INC0aDjAP19Dm1+B3sH6z9HGo1VeIyc3pn azcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=czCk0DRp2OOPFajQ4NWb1LoJtE/xpLksVA/HZQ+botI=; b=nzu8D0uAfm0wmyEFnL0ce92vVzk2Jt+AMNvhtK1oGoYpCAxxreU08FxDLOOgjZ+kOJ GXyPNV/Uz3bDZjSbW5ReudLcRI0WOlAgWuiJZLJQKcWJeRya62B7N5VMY5fuaTH+7wlQ m1zcb3KP+Rzx+A7vfIA+DxD0dCT/rcDw4crybBj2AOITGzeSe1LSEEXRGyJSzah+ujQg lwFleH+aPlrwOE9iYYz41AZcDVOKQHCQQGQZHy86ZOXDp4R4BFaPfa4TprAD/ip57kzN ZQtEcdiT3eEJ+FmtnPPI5ipYKmJgjhxg28jaHsRlVE1P3XfQp9lGSyCdcwZypWB9gOpq zmxA== X-Gm-Message-State: APjAAAXjdP2Vg26MYCF7kYjZZmjZ0G3V5phCc83fxzU+ozsLa1lJMNlI cAnyKJgVM29OlfT/nUXhDapyT4Dk X-Google-Smtp-Source: APXvYqxtNIoLx/btKk/afBEzs2rOtjE3dpg6SE8cUsSXnbbgTK5KlOgw8dsupXmTmBV7m+QRvHhrlA== X-Received: by 2002:a5d:6901:: with SMTP id t1mr20507910wru.94.1575740870349; Sat, 07 Dec 2019 09:47:50 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x11sm3237066wmg.46.2019.12.07.09.47.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:49 -0800 (PST) Message-Id: In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:35 +0000 Subject: [PATCH v5 07/15] git-p4: add new support function gitConfigSet() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene Add a new method gitConfigSet(). This method will set a value in the git configuration cache list. Signed-off-by: Ben Keene --- git-p4.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/git-p4.py b/git-p4.py index e7c24817ad..e020958083 100755 --- a/git-p4.py +++ b/git-p4.py @@ -860,6 +860,11 @@ def gitConfigList(key): _gitConfig[key] = [] return _gitConfig[key] +def gitConfigSet(key, value): + """ Set the git configuration key 'key' to 'value' for this session + """ + _gitConfig[key] = value + def p4BranchesInGit(branchesAreInRemotes=True): """Find all the branches whose names start with "p4/", looking in remotes or heads as specified by the argument. Return From patchwork Sat Dec 7 17:47:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277705 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7F38C159A for ; Sat, 7 Dec 2019 17:48:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5DC7F217BA for ; Sat, 7 Dec 2019 17:48:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="n8bzyDNA" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726654AbfLGRr6 (ORCPT ); Sat, 7 Dec 2019 12:47:58 -0500 Received: from mail-wm1-f65.google.com ([209.85.128.65]:50619 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726575AbfLGRry (ORCPT ); Sat, 7 Dec 2019 12:47:54 -0500 Received: by mail-wm1-f65.google.com with SMTP id p9so11184417wmg.0 for ; Sat, 07 Dec 2019 09:47:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ahJTPDk4z+xDr33bIFB3r9fymg4vPCK/dIbVmQ27Q70=; b=n8bzyDNAxQn0PuwUM8TEFDbQClhcs9q8uFCP/CK3/IZRby2XDVFbOSg/RS8Ek+DjJq /OnPA9mq0FKFUeGFL7hSnPlkUBjALGhyxsaVC3kBQ6ai2ttv/b5d675YwbzbhSJYmUht sV36iQTazCtNasdmiy9S6ieGgOKNjXEbfvaMswwDH/hei0A+1x6UIudu1p7wqw+qHWoe ZZ4OHnIMduihIA9JNIXzGvGZF7/Ppo+2GDUjEJlVh3CGXSN5i+DeG5dlurGdYJ/WM987 PZD4QBP8Wb8kA+iCqHLWOInHnN4kT4pkpXEF5v6GtVfI3uJafWkXCCeuMSAvSRWmpMju bARw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ahJTPDk4z+xDr33bIFB3r9fymg4vPCK/dIbVmQ27Q70=; b=DNZmuNy9vPzNu+WihI+5rne8tXsZ6nk+6ZZwJ5OvtICJXuTgVKiWN/qXJmANg/WqQC aj5I1XxprCIP01vUZbJ7lHU2qvYLTVpQOg4sdMWz0AJ1MiApAe/bzDm+Xbv26ZISV8e5 ugHFmbpSHYTHtBs4P820Wse4qf2ZmGWSx8+PKQKQe9us+4jSxzjdXaygidPQjOoBsR37 GdYHLJPo1Z7H6Pge5n1pkzNrBVJfKk1C9QpJNKF97Fmc4BmgS/PDGIzB6JNiE8Y7rlcJ UdadJp8Dt9pDbQSLd9riBX3SseqJCNrG7smwRbMv3Z4nChhF7gTFyuz2UUIcGAGaSc3P zrcQ== X-Gm-Message-State: APjAAAW2e1DYW9IKk1Y4NFgDlRhOk1hweweg/C430slV42OhszlWZ2X1 WmhOFnuF1c3Lr1qmDSysCjiCOE0U X-Google-Smtp-Source: APXvYqxwG+t4djxaJzIs/yj6/UoJ4b5LCs9f/HxtQpKhGVo0I33PINpBs+n7liNVEQyMmJ5yjw1yJQ== X-Received: by 2002:a1c:9e49:: with SMTP id h70mr17566212wme.79.1575740870949; Sat, 07 Dec 2019 09:47:50 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z8sm20402616wrq.22.2019.12.07.09.47.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:50 -0800 (PST) Message-Id: <1e677781d2cc75371b5362c7e63ea5ddf824d5da.1575740863.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:36 +0000 Subject: [PATCH v5 08/15] git-p4: add casting helper functions for python 3 conversion Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene Python 3 handles strings differently than Python 2.7. Since Python 2 is reaching it's end of life, a series of changes are being submitted to enable python 3.5 and following support. The current code fails basic tests under python 3.5. Change the existing unicode test add new support functions for Python 2 - Python 3 support. Define the following variables: - isunicode - a boolean variable that states if the version of python natively supports unicode (true) or not (false). This is true for Python 3 and false for Python 2. - unicode - a type alias for the datatype that holds a unicode string. It is assigned to a str under Python 3 and the unicode type for Python 2. - bytes - a type alias for an array of bytes. It is assigned the native bytes type for Python 3 and str for Python 2. Add the following new functions: - as_string(text) - A new function that will convert a byte array to a unicode (UTF-8) string under Python 3. Under Python 2, this returns the string unchanged. - as_bytes(text) - A new function that will convert a unicode string to a byte array under Python 3. Under Python 2, this returns the string unchanged. - to_unicode(text) - Converts a text string as Unicode(UTF-8) on both Python 2 and Python 3. Add a new function alias raw_input: If raw_input does not exist (it was renamed to input in Python 3) alias input as raw_input. The as_string() and as_bytes() functions allow for modifying the code with a minimal amount of impact on Python 2 support. When a string is expected, the as_string() will be used to "cast" the incoming "bytes" to a string type. Conversely as_bytes() will be used to cast a "string" to a "byte array" type. Since Python 2 overloads the datatype 'str' to serve both purposes, the Python 2 versions of these function do not change the data. This reduces the regression impact of these code changes. 'basestring' is removed since its only references are found in tests that were changed in modified in previous commits. Signed-off-by: Ben Keene --- git-p4.py | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 74 insertions(+), 6 deletions(-) diff --git a/git-p4.py b/git-p4.py index e020958083..e6f7513384 100755 --- a/git-p4.py +++ b/git-p4.py @@ -32,16 +32,84 @@ unicode = unicode except NameError: # 'unicode' is undefined, must be Python 3 - str = str + # + # For Python 3 which is natively unicode, we will use + # unicode for internal information but all P4 Data + # will remain in bytes + isunicode = True unicode = str bytes = bytes - basestring = (str,bytes) + + def as_string(text): + """ Return a byte array as a unicode string + """ + if text is None: + return None + if isinstance(text, bytes): + return unicode(text, "utf-8") + else: + return text + + def as_bytes(text): + """ Return a Unicode string as a byte array + """ + if text is None: + return None + if isinstance(text, bytes): + return text + else: + return bytes(text, "utf-8") + + def to_unicode(text): + """ Return a byte array as a unicode string + """ + return as_string(text) + + def path_as_string(path): + """ Converts a path to the UTF8 encoded string + """ + if isinstance(path, unicode): + return path + return encodeWithUTF8(path).decode('utf-8') + else: # 'unicode' exists, must be Python 2 - str = str + # + # We will treat the data as: + # str -> str + # bytes -> str + # So for Python 2 these functions are no-ops + # and will leave the data in the ambiguious + # string/bytes state + isunicode = False unicode = unicode bytes = str - basestring = basestring + + def as_string(text): + """ Return text unaltered (for Python 3 support) + """ + return text + + def as_bytes(text): + """ Return text unaltered (for Python 3 support) + """ + return text + + def to_unicode(text): + """ Return a string as a unicode string + """ + return text.decode('utf-8') + + def path_as_string(path): + """ Converts a path to the UTF8 encoded bytes + """ + return encodeWithUTF8(path) + +# Check for raw_input support +try: + raw_input +except NameError: + raw_input = input try: from subprocess import CalledProcessError @@ -740,7 +808,7 @@ def p4Where(depotPath): if data[:space] == depotPath: output = entry break - if output == None: + if output is None: return "" if output["code"] == "error": return "" @@ -4175,7 +4243,7 @@ def main(): global verbose verbose = cmd.verbose if cmd.needsGit: - if cmd.gitdir == None: + if cmd.gitdir is None: cmd.gitdir = os.path.abspath(".git") if not isValidGitDir(cmd.gitdir): # "rev-parse --git-dir" without arguments will try $PWD/.git From patchwork Sat Dec 7 17:47:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277707 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AA7F9186D for ; Sat, 7 Dec 2019 17:48:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 88AF3217BA for ; Sat, 7 Dec 2019 17:48:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RRuqO2bi" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726661AbfLGRr7 (ORCPT ); Sat, 7 Dec 2019 12:47:59 -0500 Received: from mail-wm1-f47.google.com ([209.85.128.47]:40805 "EHLO mail-wm1-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726579AbfLGRry (ORCPT ); Sat, 7 Dec 2019 12:47:54 -0500 Received: by mail-wm1-f47.google.com with SMTP id t14so10539164wmi.5 for ; Sat, 07 Dec 2019 09:47:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=udxcfKZlkTcXFsmoVhUqXRiSTeFNwO86ZLi7Q3Qvs7M=; b=RRuqO2bi6j+AgT/PGE1hEukm5/l/jrGFUtm/6y78cC+qKbsXRR98ceMvv0jMw13jjx WG0rzry+LWnpQLxY2PY/1R150yOl3Yk9K9Hieim/WY1VIyWDZMJoHhDaaxmc4iBiNOGn GxNC0tBC6JvauXaUhYYX+6RCaBYTYqe8rvHR7Cokd59pRzbeGIGl+KT5uenMT/o1iaWg /hUoScmZskd30GxweR9OZA4jRTSjrTzMfJlbNQjNxjgdNAt3OGy87q0FbfFcyYKVjH3T vL9I/RPmMMrYeEmRJ1MeNYiAakxRO7GP/QM095XoBo6h96slFZ1yZmzn0cFO5BvZBm5s ilLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=udxcfKZlkTcXFsmoVhUqXRiSTeFNwO86ZLi7Q3Qvs7M=; b=Dpla0aYuPYa4lai17+NV2rVrK9hKCGbAkx223O5bsMA4Mz0iBNJPJSSZMIWQC/I3V3 OmtXb0cNIEwPbYs5fSbkCdIDgpO91HUepzYub2DRKflyc5uMHvmbGVIPR7r3xWkCDw/N QOj7kdSFNJ07h/obZ2KX7wlQ8NGyUbdiYNCrUuu6thqjoe6JUky9UtP01x9WWhxOjNFK a0B3ttEu7ukUm16qWEEtLTrFtnl4mOtJjSxHbqPfD98ZPKPk84NySng6cfzL/4h5jVy2 8rJO3MA9Y1eialD7m8A5EuKz7fzyy+fyAQR1v6S+rIO7R8xOhjOcRwXHnE8pUrbTij7U fr4g== X-Gm-Message-State: APjAAAUbc5/jlJI4aiX+mojdu0SqT4YfZ37LYng6EUxHx+VdKIl83332 s0pmD7+3icMAen4zDY1OpjqBVqoj X-Google-Smtp-Source: APXvYqwZixmBefbZcw1tUw8XLQWraV+1iRn5AaeqnlOF383xqqEZscbbooLM9Hf6shckA+uGCux1gg== X-Received: by 2002:a1c:5448:: with SMTP id p8mr16778063wmi.70.1575740871716; Sat, 07 Dec 2019 09:47:51 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c1sm20259426wrs.24.2019.12.07.09.47.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:51 -0800 (PST) Message-Id: In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:37 +0000 Subject: [PATCH v5 09/15] git-p4: python 3 syntax changes Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene Python 3 handles strings differently than Python 2.7. Since Python 2 is reaching it's end of life, a series of changes are being submitted to enable python 3.5 and following support. The current code fails basic tests under python 3.5. There are a number of translations suggested by modernize/futureize that should be taken to fix numerous non-string specific issues. Change references to the X.next() iterator to the function next(X) which is compatible with both Python2 and Python3. Change references to X.keys() to list(X.keys()) to return a list that can be iterated in both Python2 and Python3. Add the literal text (object) to the end of class definitions to be consistent with Python3 class definition. Change integer divison to use "//" instead of "/" Under Both Python 2 and Python 3 // will return a floor()ed result which matches existing functionality. Change the format string for displaying decimal values from %d to %4.1f% when displaying a progress. This avoids displaying long repeating decimals in user displayed text. Signed-off-by: Ben Keene --- git-p4.py | 55 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 29 insertions(+), 26 deletions(-) diff --git a/git-p4.py b/git-p4.py index e6f7513384..fc6c9406c2 100755 --- a/git-p4.py +++ b/git-p4.py @@ -26,6 +26,9 @@ import zlib import ctypes import errno +import os.path +import codecs +import io # support basestring in Python 3 try: @@ -639,7 +642,7 @@ def parseDiffTreeEntry(entry): If the pattern is not matched, None is returned.""" - match = diffTreePattern().next().match(entry) + match = next(diffTreePattern()).match(entry) if match: return { 'src_mode': match.group(1), @@ -980,7 +983,7 @@ def findUpstreamBranchPoint(head = "HEAD"): branches = p4BranchesInGit() # map from depot-path to branch name branchByDepotPath = {} - for branch in branches.keys(): + for branch in list(branches.keys()): tip = branches[branch] log = extractLogMessageFromGitCommit(tip) settings = extractSettingsGitLog(log) @@ -1174,7 +1177,7 @@ def getClientSpec(): client_name = entry["Client"] # just the keys that start with "View" - view_keys = [ k for k in entry.keys() if k.startswith("View") ] + view_keys = [ k for k in list(entry.keys()) if k.startswith("View") ] # hold this new View view = View(client_name) @@ -1416,7 +1419,7 @@ def processContent(self, git_mode, relPath, contents): else: return LargeFileSystem.processContent(self, git_mode, relPath, contents) -class Command: +class Command(object): delete_actions = ( "delete", "move/delete", "purge" ) add_actions = ( "add", "branch", "move/add" ) @@ -1431,7 +1434,7 @@ def ensure_value(self, attr, value): setattr(self, attr, value) return getattr(self, attr) -class P4UserMap: +class P4UserMap(object): def __init__(self): self.userMapFromPerforceServer = False self.myP4UserId = None @@ -1482,7 +1485,7 @@ def getUserMapFromPerforceServer(self): self.emails[email] = user s = '' - for (key, val) in self.users.items(): + for (key, val) in list(self.users.items()): s += "%s\t%s\n" % (key.expandtabs(1), val.expandtabs(1)) open(self.getUserCacheFilename(), "wb").write(s) @@ -1833,7 +1836,7 @@ def prepareSubmitTemplate(self, changelist=None): break if not change_entry: die('Failed to decode output of p4 change -o') - for key, value in change_entry.iteritems(): + for key, value in list(change_entry.items()): if key.startswith('File'): if 'depot-paths' in settings: if not [p for p in settings['depot-paths'] @@ -2077,7 +2080,7 @@ def applyCommit(self, id): p4_delete(f) # Set/clear executable bits - for f in filesToChangeExecBit.keys(): + for f in list(filesToChangeExecBit.keys()): mode = filesToChangeExecBit[f] setP4ExecBit(f, mode) @@ -2330,7 +2333,7 @@ def run(self, args): self.clientSpecDirs = getClientSpec() # Check for the existence of P4 branches - branchesDetected = (len(p4BranchesInGit().keys()) > 1) + branchesDetected = (len(list(p4BranchesInGit().keys())) > 1) if self.useClientSpec and not branchesDetected: # all files are relative to the client spec @@ -2721,7 +2724,7 @@ def __init__(self): self.knownBranches = {} self.initialParents = {} - self.tz = "%+03d%02d" % (- time.timezone / 3600, ((- time.timezone % 3600) / 60)) + self.tz = "%+03d%02d" % (- time.timezone // 3600, ((- time.timezone % 3600) // 60)) self.labels = {} # Force a checkpoint in fast-import and wait for it to finish @@ -2838,7 +2841,7 @@ def splitFilesIntoBranches(self, commit): else: relPath = self.stripRepoPath(path, self.depotPaths) - for branch in self.knownBranches.keys(): + for branch in list(self.knownBranches.keys()): # add a trailing slash so that a commit into qt/4.2foo # doesn't end up in qt/4.2, e.g. if p4PathStartsWith(relPath, branch + "/"): @@ -2867,7 +2870,7 @@ def streamOneP4File(self, file, contents): size = int(self.stream_file['fileSize']) else: size = 0 # deleted files don't get a fileSize apparently - sys.stdout.write('\r%s --> %s (%i MB)\n' % (file['depotFile'], relPath, size/1024/1024)) + sys.stdout.write('\r%s --> %s (%i MB)\n' % (file['depotFile'], relPath, size//1024//1024)) sys.stdout.flush() (type_base, type_mods) = split_p4_type(file["type"]) @@ -2967,7 +2970,7 @@ def streamP4FilesCb(self, marshalled): required_bytes = int((4 * int(self.stream_file["fileSize"])) - calcDiskFree()) if required_bytes > 0: err = 'Not enough space left on %s! Free at least %i MB.' % ( - os.getcwd(), required_bytes/1024/1024 + os.getcwd(), required_bytes//1024//1024 ) if err: @@ -2996,7 +2999,7 @@ def streamP4FilesCb(self, marshalled): # pick up the new file information... for the # 'data' field we need to append to our array - for k in marshalled.keys(): + for k in list(marshalled.keys()): if k == 'data': if 'streamContentSize' not in self.stream_file: self.stream_file['streamContentSize'] = 0 @@ -3011,8 +3014,8 @@ def streamP4FilesCb(self, marshalled): 'depotFile' in self.stream_file): size = int(self.stream_file["fileSize"]) if size > 0: - progress = 100*self.stream_file['streamContentSize']/size - sys.stdout.write('\r%s %d%% (%i MB)' % (self.stream_file['depotFile'], progress, int(size/1024/1024))) + progress = 100.0*self.stream_file['streamContentSize']/size + sys.stdout.write('\r%s %4.1f%% (%i MB)' % (self.stream_file['depotFile'], progress, int(size//1024//1024))) sys.stdout.flush() self.stream_have_file_info = True @@ -3093,7 +3096,7 @@ def streamTag(self, gitStream, labelName, labelDetails, commit, epoch): gitStream.write("tagger %s\n" % tagger) - print("labelDetails=",labelDetails) + print(("labelDetails=",labelDetails)) if 'Description' in labelDetails: description = labelDetails['Description'] else: @@ -3232,7 +3235,7 @@ def getLabels(self): self.labels[newestChange] = [output, revisions] if self.verbose: - print("Label changes: %s" % self.labels.keys()) + print("Label changes: %s" % list(self.labels.keys())) # Import p4 labels as git tags. A direct mapping does not # exist, so assume that if all the files are at the same revision @@ -3375,7 +3378,7 @@ def getBranchMapping(self): def getBranchMappingFromGitBranches(self): branches = p4BranchesInGit(self.importIntoRemotes) - for branch in branches.keys(): + for branch in list(branches.keys()): if branch == "master": branch = "main" else: @@ -3487,14 +3490,14 @@ def importChanges(self, changes, origin_revision=0): self.updateOptionDict(description) if not self.silent: - sys.stdout.write("\rImporting revision %s (%s%%)" % (change, cnt * 100 / len(changes))) + sys.stdout.write("\rImporting revision %s (%4.1f%%)" % (change, cnt * 100 / len(changes))) sys.stdout.flush() cnt = cnt + 1 try: if self.detectBranches: branches = self.splitFilesIntoBranches(description) - for branch in branches.keys(): + for branch in list(branches.keys()): ## HACK --hwn branchPrefix = self.depotPaths[0] + branch + "/" self.branchPrefixes = [ branchPrefix ] @@ -3683,13 +3686,13 @@ def run(self, args): if short in branches: self.p4BranchesInGit = [ short ] else: - self.p4BranchesInGit = branches.keys() + self.p4BranchesInGit = list(branches.keys()) if len(self.p4BranchesInGit) > 1: if not self.silent: print("Importing from/into multiple branches") self.detectBranches = True - for branch in branches.keys(): + for branch in list(branches.keys()): self.initialParents[self.refPrefix + branch] = \ branches[branch] @@ -4073,7 +4076,7 @@ def findLastP4Revision(self, starting_point): to find the P4 commit we are based on, and the depot-paths. """ - for parent in (range(65535)): + for parent in (list(range(65535))): log = extractLogMessageFromGitCommit("{0}^{1}".format(starting_point, parent)) settings = extractSettingsGitLog(log) if 'change' in settings: @@ -4212,7 +4215,7 @@ def printUsage(commands): def main(): if len(sys.argv[1:]) == 0: - printUsage(commands.keys()) + printUsage(list(commands.keys())) sys.exit(2) cmdName = sys.argv[1] @@ -4222,7 +4225,7 @@ def main(): except KeyError: print("unknown command %s" % cmdName) print("") - printUsage(commands.keys()) + printUsage(list(commands.keys())) sys.exit(2) options = cmd.options From patchwork Sat Dec 7 17:47:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277709 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D70D517EF for ; Sat, 7 Dec 2019 17:48:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B33CD217BA for ; Sat, 7 Dec 2019 17:48:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Xo5FW7mD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726667AbfLGRsA (ORCPT ); Sat, 7 Dec 2019 12:48:00 -0500 Received: from mail-wr1-f68.google.com ([209.85.221.68]:36355 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726595AbfLGRrx (ORCPT ); Sat, 7 Dec 2019 12:47:53 -0500 Received: by mail-wr1-f68.google.com with SMTP id z3so11345686wru.3 for ; Sat, 07 Dec 2019 09:47:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Z3oyfxleLGsd3u4DfBh7GyxAd22AsIddLqrP+H0OSzE=; b=Xo5FW7mDsxR9YuYw97+peyEfobwyPQkHDEwclPmHHLGs8D2+5GbeC+eko5zT0oU0Bu 9SiUOk632xEJx1TgNZqE0KImdPIGRtVb07uf30MrUgwEC0SVdI5qKP1SluesKgOvEHhA W24rMeQ5bMEli2Rzfs8lvOEOT7C50KvR3LLXWaza0PbBgO/tFNyX/yfqqvf1ag1Qk7eM fp0eddRioha7TMmAWKThwgpKSR4uFI2r2lfy0ow43qaunhbvff2v4i5xvAWNk1RfA2RD MfmAnnnC5HholY8tcZLDSWXzDtPkbhpo1PCbn7oloQnRhDPTmleZezKI+kSZsmAlM2vv V58Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Z3oyfxleLGsd3u4DfBh7GyxAd22AsIddLqrP+H0OSzE=; b=j6jvOxq4bKjvt5EEHL4ZUnxzBF65+sUOQcouNmD1/RyPSXvVncs+7m/ybyzk0iWeu0 t1z1vBEOCc08TcnvWhlM7/3ODqEsyayMlu50hHUBy5K9hMElcg3HhR03awCPorynnMs6 wjh25FBrkX9WWWNyOtHII9zAfkMCGgszLaZ5tsRdqtZ8tQL4oiuUFMzZCMOzAhvrs1px KYlLJWHHuxxNL+I9SuNYWNlJWQVbZkBrXVJANQeXwKxh68RKlLJTomX65p6j2LxDHsT/ tubsQyl6YPavs6Hr+f3T/s/VRaMJeIeGTuJj1+z43+7tcREJOwn0ZO/DhzFtp/Qp0GW/ 92xw== X-Gm-Message-State: APjAAAWp3lAd85sqMdpmlsiIWBL15TojjG4DwvbRwPwRtYzNTCDVMttw xoEiu5EiQgj2o7nsLosWuTFpe6I7 X-Google-Smtp-Source: APXvYqwq3Wvb2ivQ2behwHreZCD1EaTTdE6x4/CfuCinKe4o4LZ/VRAYJChZkIn28Mh9z9F3I3QeEQ== X-Received: by 2002:adf:f20b:: with SMTP id p11mr21243310wro.195.1575740872513; Sat, 07 Dec 2019 09:47:52 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a16sm19791993wrt.37.2019.12.07.09.47.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:51 -0800 (PST) Message-Id: In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:38 +0000 Subject: [PATCH v5 10/15] git-p4: fix assumed path separators to be more Windows friendly Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene When a computer is configured to use Git for windows and Python for windows, and not a Unix subsystem like cygwin or WSL, the directory separator changes and causes git-p4 to fail to properly determine paths. Fix 3 path separator errors: 1. getUserCacheFilename() - should not use string concatenation. Change this code to use os.path.join to build an OS tolerant path. 2. defaultDestiantion used the OS.path.split to split depot paths. This is incorrect on windows. Change the code to split on a forward slash(/) instead since depot paths use this character regardless of the operating system. 3. The call to isValidGitDir() in the main code also used a literal forward slash. Change the code to use os.path.join to correctly format the path for the operating system. These three changes allow the suggested windows configuration to properly locate files while retaining the existing behavior on non-windows operating systems. Signed-off-by: Ben Keene --- git-p4.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/git-p4.py b/git-p4.py index fc6c9406c2..1838045078 100755 --- a/git-p4.py +++ b/git-p4.py @@ -1459,8 +1459,10 @@ def p4UserIsMe(self, p4User): return True def getUserCacheFilename(self): + """ Returns the filename of the username cache + """ home = os.environ.get("HOME", os.environ.get("USERPROFILE")) - return home + "/.gitp4-usercache.txt" + return os.path.join(home, ".gitp4-usercache.txt") def getUserMapFromPerforceServer(self): if self.userMapFromPerforceServer: @@ -3978,13 +3980,16 @@ def __init__(self): self.cloneBare = False def defaultDestination(self, args): + """ Returns the last path component as the default git + repository directory name + """ ## TODO: use common prefix of args? depotPath = args[0] depotDir = re.sub("(@[^@]*)$", "", depotPath) depotDir = re.sub("(#[^#]*)$", "", depotDir) depotDir = re.sub(r"\.\.\.$", "", depotDir) depotDir = re.sub(r"/$", "", depotDir) - return os.path.split(depotDir)[1] + return depotDir.split('/')[-1] def run(self, args): if len(args) < 1: @@ -4257,8 +4262,8 @@ def main(): chdir(cdup); if not isValidGitDir(cmd.gitdir): - if isValidGitDir(cmd.gitdir + "/.git"): - cmd.gitdir += "/.git" + if isValidGitDir(os.path.join(cmd.gitdir, ".git")): + cmd.gitdir = os.path.join(cmd.gitdir, ".git") else: die("fatal: cannot locate git repository at %s" % cmd.gitdir) From patchwork Sat Dec 7 17:47:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277703 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 545A8139A for ; Sat, 7 Dec 2019 17:48:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 32FB4217BA for ; Sat, 7 Dec 2019 17:48:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="J0Qp+qbT" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726643AbfLGRr5 (ORCPT ); Sat, 7 Dec 2019 12:47:57 -0500 Received: from mail-wr1-f41.google.com ([209.85.221.41]:41635 "EHLO mail-wr1-f41.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726619AbfLGRrz (ORCPT ); Sat, 7 Dec 2019 12:47:55 -0500 Received: by mail-wr1-f41.google.com with SMTP id c9so11318233wrw.8 for ; Sat, 07 Dec 2019 09:47:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=OS5efDlzdZteNOo7RSoCHPJVXI1UrPtxfmlVrDV6KqQ=; b=J0Qp+qbT69WETi+baOibqnO4+6X5W1A7Uv1l6pnSrtIpVerSYNqDp4MTor5zfNVZOJ QxOU0zlJ3ugc6uvBin9Xb2kN7nBTxF7pcmb6pA+9qOOW52sceaXsPq1hQ7RSmG4oYa59 bXRNDwFPBfIfFjSE+FVtNgwW8+lUbU9DDtbPBiPF5cUKnLivD9Nyqizu06JJ3jUiplwT H5bEPq0Amt+8/ZBgTfLS/oeu7GqJLYs/SOsxGCQzO5nE7fL6OtzCaXxmP1F9TGGexEqk WHEcr33XaL4+cxEv102dQo6cVvEADYcX8Qj1YY/ALpoqw1vE36/YH5yUoekrdDeCqfec J1dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=OS5efDlzdZteNOo7RSoCHPJVXI1UrPtxfmlVrDV6KqQ=; b=CfWJXT+UVZDgqWtlvBrTLYA9ds6nyXN8J+rDUb4hIScPwoygKPzRRojDguIEXXwVQq jYREFz7EHf9Dj1kWe1ABPoWEZOoPj1tWQXYVRilnAaYIyuQfEaUCZfhWJdFkuPi46x7K miLPt3ZCGw93BFrWwMo0xwlF9Vlzmgih+51bsi3CiXOu6qpzxqqz4bZBk1EkyMo8hyUw HmIUZyIT5YvgA8uPFJD2Ke6V/6Q2m/SH1hDyWbRjURf+590528i7qtdDFfuWLwZYRNcj w+FVWZJxIu0FpCXu3M+k/bycZDmRzsM47ZmBQqVuoCdBWxox51C40aXQRjAaGFLys7PC VCkA== X-Gm-Message-State: APjAAAVE7KJMQ46MCaEByCfIBANfe0WavHi2lCBi1pBgZQZO8lLTVwFm VzwwcGs8klgSePilF6ThB8JdxL5h X-Google-Smtp-Source: APXvYqwamq5KdV2Tw/yJS0eS+QZA59GHwGTe8K+yx+sxvFDI6XPCbQoQxDsNoCfmZHN99YHZMzfwjA== X-Received: by 2002:a5d:66c3:: with SMTP id k3mr20607497wrw.370.1575740873263; Sat, 07 Dec 2019 09:47:53 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l13sm7322482wmh.12.2019.12.07.09.47.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:52 -0800 (PST) Message-Id: In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:39 +0000 Subject: [PATCH v5 11/15] git-p4: add Py23File() - helper class for stream writing Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene This is a preparatory commit that does not change current behavior. It adds a new class Py23File. Following the Python recommendation of keeping text as unicode internally and only converting to and from bytes on input and output, this class provides an interface for the methods used for reading and writing files and file like streams. A new class was implemented to avoid requiring additional dependencies. Create a class that wraps the input and output functions used by the git-p4.py code for reading and writing to standard file handles. The methods of this class should take a Unicode string for writing and return unicode strings in reads. This class should be a drop-in for existing file like streams The following methods should be coded for supporting existing read/write calls: * write - this should write a Unicode string to the underlying stream * read - this should read from the underlying stream and cast the bytes as a unicode string * readline - this should read one line of text from the underlying stream and cast it as a unicode string * readline - this should read a number of lines, optionally hinted, and cast each line as a unicode string The expression "cast as a unicode string" is used because the code should use the as_bytes() and as_string() functions instead of cohercing the data to actual unicode strings or bytes. This allows Python 2 code to continue to use the internal "str" data type instead of converting the data back and forth to actual unicode strings. This retains current Python 2 support while Python 3 support may be incomplete. Signed-off-by: Ben Keene --- git-p4.py | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/git-p4.py b/git-p4.py index 1838045078..03829f796d 100755 --- a/git-p4.py +++ b/git-p4.py @@ -4187,6 +4187,72 @@ def run(self, args): print("%s <= %s (%s)" % (branch, ",".join(settings["depot-paths"]), settings["change"])) return True +class Py23File(): + """ Python2/3 Unicode File Wrapper + """ + + stream_handle = None + verbose = False + debug_handle = None + + def __init__(self, stream_handle, verbose = False): + """ Create a Python3 compliant Unicode to Byte String + Windows compatible wrapper + + stream_handle = the underlying file-like handle + verbose = Boolean if content should be echoed + """ + self.stream_handle = stream_handle + self.verbose = verbose + + def write(self, utf8string): + """ Writes the utf8 encoded string to the underlying + file stream + """ + self.stream_handle.write(as_bytes(utf8string)) + if self.verbose: + sys.stderr.write("Stream Output: %s" % utf8string) + sys.stderr.flush() + + def read(self, size = None): + """ Reads int charcters from the underlying stream + and converts it to utf8. + + Be aware, the size value is for reading the underlying + bytes so the value may be incorrect. Usage of the size + value is discouraged. + """ + if size == None: + return as_string(self.stream_handle.read()) + else: + return as_string(self.stream_handle.read(size)) + + def readline(self): + """ Reads a line from the underlying byte stream + and converts it to utf8 + """ + return as_string(self.stream_handle.readline()) + + def readlines(self, sizeHint = None): + """ Returns a list containing lines from the file converted to unicode. + + sizehint - Optional. If the optional sizehint argument is + present, instead of reading up to EOF, whole lines totalling + approximately sizehint bytes are read. + """ + lines = self.stream_handle.readlines(sizeHint) + for i in range(0, len(lines)): + lines[i] = as_string(lines[i]) + return lines + + def close(self): + """ Closes the underlying byte stream """ + self.stream_handle.close() + + def flush(self): + """ Flushes the underlying byte stream """ + self.stream_handle.flush() + class HelpFormatter(optparse.IndentedHelpFormatter): def __init__(self): optparse.IndentedHelpFormatter.__init__(self) From patchwork Sat Dec 7 17:47:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277721 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7CAD7139A for ; Sat, 7 Dec 2019 17:48:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5215C24677 for ; Sat, 7 Dec 2019 17:48:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nsJZA5Oy" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726684AbfLGRsG (ORCPT ); Sat, 7 Dec 2019 12:48:06 -0500 Received: from mail-wm1-f42.google.com ([209.85.128.42]:51304 "EHLO mail-wm1-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726627AbfLGRr4 (ORCPT ); Sat, 7 Dec 2019 12:47:56 -0500 Received: by mail-wm1-f42.google.com with SMTP id g206so11158851wme.1 for ; Sat, 07 Dec 2019 09:47:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=fqOeCHbhS1rw+kBmqBgQZVR8txMGwT35WxogXmqsgMQ=; b=nsJZA5Oy9ut5LR17zV4VjYfTR5Iu+yHt1h7bD2Up0A1Px65+hTxKTWqFKwchcxBaJi XYCNQJokSiqlVtgdIz9qIEpL4u4E3nCdv9IJ4Nw5a82VQm65OXiJVkKPaN2yHAS8lRGT xajgJptqevvHIz68wlj1tQcWB2ZfzAitcNyKDHPAt7HIIqNltzEopte6l51MG52Xxck8 lnX66RCKO7oXNsD+Cnbbl9MgAAC7xohmpIvxA+jmP4zbLM9EVKxc1ca5aq1WXkTsHm6b TNlGJqrlrF0/qVQpoRn+kYlb/OshAMCikmokkAezsxo0zfT6Smnhd9cR8ObvgtX35rMH K5bQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=fqOeCHbhS1rw+kBmqBgQZVR8txMGwT35WxogXmqsgMQ=; b=CpuE/LR4MQqlNSdo/xWWCyl/VFFiAlF8j1b3o1O4aHbxAEsGK1lCXRxDlrkwAB5rGM GMgNCb7Beulx/AM+H8f+LeVKdAG7s2tHHQG0Fytk5cEE4qwD2eP2ctHd891EJ9Gedk0r iIroSN0d6o9LjFnL7ScAtAwPnJXfZwgZlQ6siH44e6YbJTdDBG29VeumVBWNeCx6O1eE hACduYygwitlN8NLpAqBpB6op370Ax5AXDDOX/PJIuhyFYh33UhbrH8w0vtLK1sZ5fJR QfGG+Kf6kfNz0jSwlFPpexR4bpQ3utI7b+/68ZXeepyjYFeHXNEMsEnrIhtWWyiwymiu 9zDw== X-Gm-Message-State: APjAAAXwr8r7deoyndCi6Ot4uTORd4cKH+OXK7lx2qHhslXcknHuGbeO NM3/vzgCcrSr93ybwhD+DWUj5Min X-Google-Smtp-Source: APXvYqzMMDCzoCi0C6y2Ki3D8jr6g1Sg9mLyDCIt3ZrtlKs27vlVAHcGtWvZNNNdCMbdP1rsPV8KEg== X-Received: by 2002:a05:600c:2409:: with SMTP id 9mr15381327wmp.109.1575740873972; Sat, 07 Dec 2019 09:47:53 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q3sm20697115wrn.33.2019.12.07.09.47.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:53 -0800 (PST) Message-Id: In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:40 +0000 Subject: [PATCH v5 12/15] git-p4: p4CmdList - support Unicode encoding Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene The p4CmdList is a commonly used function in the git-p4 code. It is used to execute a command in P4 and return the results of the call in a list. The problem is that p4CmdList takes bytes as the parameter data and returns bytes in the return list. Add a new optional parameter to the signature, encode_cmd_output, that determines if the dictionary values returned in the function output are treated as bytes or as strings. Change the code to conditionally pass the output data through the as_string() function when encode_cmd_output is true. Otherwise the function should return the data as bytes. Change the code so that regardless of the setting of encode_cmd_output, the dictionary keys in the return value will always be encoded with as_string(). as_string(bytes) is a method defined in this project that treats the byte data as a string. The word "string" is used because the meaning varies depending on the version of Python: - Python 2: The "bytes" are returned as "str", functionally a No-op. - Python 3: The "bytes" are returned as a Unicode string. The p4CmdList function returns a list of dictionaries that contain the result of p4 command. If the callback (cb) is defined, the standard output of the p4 command is redirected. Data that is passed to the standard input of the P4 process should be as_bytes() to avoid conversion unicode encoding errors. as_bytes(text) is a method defined in this project that treats the text data as a string that should be converted to a byte array (bytes). The behavior of this function depends on the version of python: - Python 2: The "text" is returned as "str", functionally a No-op. - Python 3: The "text" is treated as a UTF-8 encoded Unicode string and is decoded to bytes. Additionally, change literal text prior to conversion to be literal bytes for the code that is evaluating the standard output from the p4 call. Add encode_cmd_output to the p4Cmd since this is a helper function that wraps the behavior of p4CmdList. Signed-off-by: Ben Keene --- git-p4.py | 36 ++++++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 8 deletions(-) diff --git a/git-p4.py b/git-p4.py index 03829f796d..e8f31339e4 100755 --- a/git-p4.py +++ b/git-p4.py @@ -716,7 +716,23 @@ def isModeExecChanged(src_mode, dst_mode): return isModeExec(src_mode) != isModeExec(dst_mode) def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False, - errors_as_exceptions=False): + errors_as_exceptions=False, encode_cmd_output=True): + """ Executes a P4 command: 'cmd' optionally passing 'stdin' to the command's + standard input via a temporary file with 'stdin_mode' mode. + + Output from the command is optionally passed to the callback function 'cb'. + If 'cb' is None, the response from the command is parsed into a list + of resulting dictionaries. (For each block read from the process pipe.) + + If 'skip_info' is true, information in a block read that has a code type of + 'info' will be skipped. + + If 'errors_as_exceptions' is set to true (the default is false) the error + code returned from the execution will generate an exception. + + If 'encode_cmd_output' is set to true (the default) the data that is returned + by this function will be passed through the "as_string" function. + """ if not isinstance(cmd, list): cmd = "-G " + cmd @@ -739,7 +755,7 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False, stdin_file.write(stdin) else: for i in stdin: - stdin_file.write(i + '\n') + stdin_file.write(as_bytes(i) + b'\n') stdin_file.flush() stdin_file.seek(0) @@ -753,12 +769,15 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False, while True: entry = marshal.load(p4.stdout) if skip_info: - if 'code' in entry and entry['code'] == 'info': + if b'code' in entry and entry[b'code'] == b'info': continue if cb is not None: cb(entry) else: - result.append(entry) + out = {} + for key, value in entry.items(): + out[as_string(key)] = (as_string(value) if encode_cmd_output else value) + result.append(out) except EOFError: pass exitCode = p4.wait() @@ -785,8 +804,9 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False, return result -def p4Cmd(cmd): - list = p4CmdList(cmd) +def p4Cmd(cmd, encode_cmd_output=True): + """Executes a P4 command and returns the results in a dictionary""" + list = p4CmdList(cmd, encode_cmd_output=encode_cmd_output) result = {} for entry in list: result.update(entry) @@ -1165,7 +1185,7 @@ def getClientSpec(): """Look at the p4 client spec, create a View() object that contains all the mappings, and return it.""" - specList = p4CmdList("client -o") + specList = p4CmdList("client -o", encode_cmd_output=False) if len(specList) != 1: die('Output from "client -o" is %d lines, expecting 1' % len(specList)) @@ -2609,7 +2629,7 @@ def update_client_spec_path_cache(self, files): if len(fileArgs) == 0: return # All files in cache - where_result = p4CmdList(["-x", "-", "where"], stdin=fileArgs) + where_result = p4CmdList(["-x", "-", "where"], stdin=fileArgs, encode_cmd_output=False) for res in where_result: if "code" in res and res["code"] == "error": # assume error is "... file(s) not in client view" From patchwork Sat Dec 7 17:47:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277719 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A4B991805 for ; Sat, 7 Dec 2019 17:48:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 659642467A for ; Sat, 7 Dec 2019 17:48:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SDvF9Oya" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726682AbfLGRsF (ORCPT ); Sat, 7 Dec 2019 12:48:05 -0500 Received: from mail-wr1-f54.google.com ([209.85.221.54]:40167 "EHLO mail-wr1-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726469AbfLGRsA (ORCPT ); Sat, 7 Dec 2019 12:48:00 -0500 Received: by mail-wr1-f54.google.com with SMTP id c14so11305376wrn.7 for ; Sat, 07 Dec 2019 09:47:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=68Orkq5COgZJ1yjNGDITAA4+beKp29jHJfxjj5LbVlg=; b=SDvF9Oya7S9JVb2x3WhISMA+cJpS6aX0l+5uDlkP5FTlqgadaSm73ftYV6K9b0TKMl pLMFUuQT1oNY5aeoDab789SMLIzZLr/tWTL7s+8E3JKm79Iz+5o+oiKmlnfb1Ztu2c73 oM1La16Rr30HI/9PKXIyy9VEylsN7C/RNrrDTgYWrWXfPYd0paQGP8I27tBLKKhNqMSN cG+Ayn0GCuNKUjnsaKKpyKjf3GA/sTmC3byHDHtBUjxDzEjgqr6STCLTAAT9l9Z651tU yXUnF0qmy0huJ/3WDBYe9wkvYux/6oGMeEJnwhud0Ny4H6WeJOn40T9eDvIptfkAE/ga NcBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=68Orkq5COgZJ1yjNGDITAA4+beKp29jHJfxjj5LbVlg=; b=Ob0ILzk9EaWeNInvR4RX0/rLRJN42f1fbhvcOowVq6iHQarBinI8DJ7X9hG1x66Yn1 kGXf4Ts+W10neenAShoNM3GwhxYwBsJIgfoNUM88CUs/8IEtHEmXEW0WfwK4V/NKkl3j uLNopd3uWZ9tqCfRvDHxL364VTtDtIVvJ6+FODgyN/CeKHd2RBLG0teGOurIidvS8LMr v4qU7Yo3AVUsOupX5Rpdl2r8B8qUnv2t73zvocmd3GcNZxwh7aYb4aehOryEXvG1z//t 9LJeKiCjhBgFBRNspIdmUdFywATw4KNfv0wPLgjzHYfWvnC79CKmI7HvKtyAliJHVEwf o/yQ== X-Gm-Message-State: APjAAAW/OFzqN/PcHUhTjtuj8ji9Ht+FJcqDWBMCIF/UQwEpzJ+OQzHE aHSvpWv6E1SHDRMoJFLHLA7MJ8Yf X-Google-Smtp-Source: APXvYqzD6Dt+xi6mUbTDOREq6inXhtL1ZvwjzfnQVceObHxow907J6ynfe9lBh+qe4lySPJRUZ9wsA== X-Received: by 2002:adf:fe12:: with SMTP id n18mr21095561wrr.158.1575740874774; Sat, 07 Dec 2019 09:47:54 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k13sm18684019wrx.59.2019.12.07.09.47.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:54 -0800 (PST) Message-Id: In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:41 +0000 Subject: [PATCH v5 13/15] git-p4: support Python 3 for basic P4 clone, sync, and submit (t9800) Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene NOTE: Python 3 is still not properly supported for any use with the git-p4 python code. Warning - this is a very large atomic commit. The commit text is also very large. Change the code such that, with the exception of P4 depot paths and depot files, all text read by git-p4 is cast as a string as soon as possible and converted back to bytes as late as possible, following Python 2 to Python 3 conversion best practices. Important: Do not cast the bytes that contain the p4 depot path or p4 depot file name. These should be left as bytes until used. These two values should not be converted because the encoding of these values is unknown. git-p4 supports a configuration value git-p4.pathEncoding that is used by the encodeWithUTF8() to determine what a UTF8 version of the path and filename should be. However, since depot path and depot filename need to be sent to P4 in their original encoding, they will be left as byte streams until they are actually used: * When sent to P4, the bytes are literally passed to the p4 command * When displayed in text for the user, they should be passed through the path_as_string() function * When used by GIT they should be passed through the encodeWithUTF8() function Change all the rest of system calls to cast output from system calls (stdin) as_bytes() and input (stdout) as_string(). This retains existing Python 2 support, and adds python 3 support for these functions: * read_pipe_full(c) * read_pipe_lines(c) * p4_has_move_command() - used internally * gitConfig(key, typeSpecifier=None) * branch_exists(branch) * GitLFS.generatePointer(cloneDestination, contentFile) * P4Submit.applyCommit(id) - template must be read and written to the temporary file as_bytes() since it is created in memory as a string. * P4Sync.streamOneP4File(file, contents) - wrap calls to the depotFile in path_as_string() for display. The file contents must be retained as bytes, so update the RCS changes to be forced to bytes. * P4Sync.streamP4Files(marshalled) * P4Sync.importHeadRevision(revision) - encode the depotPaths for display separate from the text for processing. Py23File usage - Change the P4Sync.OpenStreams() function to cast the gitOutput, gitStream, and gitError streams as Py23File() wrapper classes. This facilitates taking strings in both python 2 and python 3 and casting them to bytes in the wrapper class instead of having to modify each method. Since the fast-import command also expects a raw byte stream for file content, add a new stream handle - gitStreamBytes which is an unwrapped verison of gitStream. Literal text - Depending on context, most literal text does not need casting to unicode or bytes as the text is Python dependent - In Python 2, the string is implied as 'str' and python 3 the string is implied as 'unicode'. Under these conditions, they match the rest of the operating text, following best practices. However, when a literal string is used in functions that are dealing with the raw input from and raw ouput to files streams, literal bytes may be required. Additionally, functions that are dealing with P4 depot paths or P4 depot file names are also dealing with bytes and will require the same casting as bytes. The following functions cast text as byte strings: * wildcard_decode(path) - the path parameter is a P4 depot and is bytes. Cast all the literals to bytes. * wildcard_encode(path) - the path parameter is a P4 depot and is bytes. Cast all the literals to bytes. * P4Sync.streamP4FilesCb(marshalled) - the marshalled data is in bytes. Cast the literals as bytes. When using this data to manipulate self.stream_file, encode all the marshalled data except for the 'depotFile' name. * P4Sync.streamP4Files(marshalled) Special behavior: * p4_describep4_describe(change, shelved=False) - encoding is disabled for the depotFile(x) and path elements since these are depot path and depo filenames. * p4PathStartsWith(path, prefix) - Since P4 depot paths can contain non-UTF-8 encoded strings, change this method to compare paths while supporting the optional encoding. - First, perform a byte-to-byte check to see if the path and prefix are both identical text. There is no need to perform encoding conversions if the text is identical. - If the byte check fails, pass both the path and prefix through encodeWithUTF8() to ensure both paths are using the same encoding. Then perform the test as originally written. * P4Submit.patchRCSKeywords(file, pattern) - the parameters of file and pattern are both strings. However this function changes the contents of the file itentified by name "file". Treat the content of this file as binary to ensure that python does not accidently change the original encoding. The regular expression is cast as_bytes() and run against the file as_bytes(). The P4 keywords are ASCII strings and cannot span lines so iterating over each line of the file is acceptable. * P4Sync.writeToGitStream(gitMode, relPath, contents) - Since 'contents' is already bytes data, instead of using the self.gitStream, use the new self.gitStreamBytes - the unwrapped gitStream that does not cast as_bytes() the binary data. * P4Sync.commit(details, files, branch, parent = "", allow_empty=False) Changed the encoding for the commit message to the preferred format for fast-import. The number of bytes is sent in the data block instead of using the EOT marker. * Change the code for handling the user cache to use binary files. Cast text as_bytes() when writing to the cache and as_string() when reading from the cache. This makes the reading and writing of the cache determinstic in it's encoding. Unlike file paths, P4 encodes the user names in UTF-8 encoding so no additional string encoding is required. Signed-off-by: Ben Keene --- git-p4.py | 285 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 203 insertions(+), 82 deletions(-) diff --git a/git-p4.py b/git-p4.py index e8f31339e4..9cf4e94e28 100755 --- a/git-p4.py +++ b/git-p4.py @@ -273,6 +273,8 @@ def read_pipe_full(c): expand = not isinstance(c, list) p = subprocess.Popen(c, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=expand) (out, err) = p.communicate() + out = as_string(out) + err = as_string(err) return (p.returncode, out, err) def read_pipe(c, ignore_error=False): @@ -299,10 +301,17 @@ def read_pipe_text(c): return out.rstrip() def p4_read_pipe(c, ignore_error=False): + """ Read output from the P4 command 'c'. Returns the output text on + success. On failure, terminates execution, unless + ignore_error is True, when it returns an empty string. + """ real_cmd = p4_build_cmd(c) return read_pipe(real_cmd, ignore_error) def read_pipe_lines(c): + """ Returns a list of text from executing the command 'c'. + The program will die if the command fails to execute. + """ if verbose: sys.stderr.write('Reading pipe: %s\n' % str(c)) @@ -312,6 +321,11 @@ def read_pipe_lines(c): val = pipe.readlines() if pipe.close() or p.wait(): die('Command failed: %s' % str(c)) + # Unicode conversion from byte-string + # Iterate and fix in-place to avoid a second list in memory. + if isunicode: + for i in range(len(val)): + val[i] = as_string(val[i]) return val @@ -340,6 +354,8 @@ def p4_has_move_command(): cmd = p4_build_cmd(["move", "-k", "@from", "@to"]) p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) (out, err) = p.communicate() + out=as_string(out) + err=as_string(err) # return code will be 1 in either case if err.find("Invalid option") >= 0: return False @@ -467,16 +483,20 @@ def p4_last_change(): return int(results[0]['change']) def p4_describe(change, shelved=False): - """Make sure it returns a valid result by checking for - the presence of field "time". Return a dict of the - results.""" + """ Returns information about the requested P4 change list. + + Data returned is not string encoded (returned as bytes) + """ + # Make sure it returns a valid result by checking for + # the presence of field "time". Return a dict of the + # results. cmd = ["describe", "-s"] if shelved: cmd += ["-S"] cmd += [str(change)] - ds = p4CmdList(cmd, skip_info=True) + ds = p4CmdList(cmd, skip_info=True, encode_cmd_output=False) if len(ds) != 1: die("p4 describe -s %d did not return 1 result: %s" % (change, str(ds))) @@ -486,12 +506,23 @@ def p4_describe(change, shelved=False): die("p4 describe -s %d exited with %d: %s" % (change, d["p4ExitCode"], str(d))) if "code" in d: - if d["code"] == "error": + if d["code"] == b"error": die("p4 describe -s %d returned error code: %s" % (change, str(d))) if "time" not in d: die("p4 describe -s %d returned no \"time\": %s" % (change, str(d))) + # Do not convert 'depotFile(X)' or 'path' to be UTF-8 encoded, however + # cast as_string() the rest of the text. + keys=d.keys() + for key in keys: + if key.startswith('depotFile'): + d[key]=d[key] + elif key == 'path': + d[key]=d[key] + else: + d[key] = as_string(d[key]) + return d # @@ -914,13 +945,15 @@ def gitDeleteRef(ref): _gitConfig = {} def gitConfig(key, typeSpecifier=None): + """ Return a configuration setting from GIT + """ if key not in _gitConfig: cmd = [ "git", "config" ] if typeSpecifier: cmd += [ typeSpecifier ] cmd += [ key ] s = read_pipe(cmd, ignore_error=True) - _gitConfig[key] = s.strip() + _gitConfig[key] = as_string(s).strip() return _gitConfig[key] def gitConfigBool(key): @@ -994,6 +1027,7 @@ def branch_exists(branch): cmd = [ "git", "rev-parse", "--symbolic", "--verify", branch ] p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) out, _ = p.communicate() + out = as_string(out) if p.returncode: return False # expect exactly one line of output: the branch name @@ -1177,9 +1211,22 @@ def p4PathStartsWith(path, prefix): # # we may or may not have a problem. If you have core.ignorecase=true, # we treat DirA and dira as the same directory + + # Since we have to deal with mixed encodings for p4 file + # paths, first perform a simple startswith check, this covers + # the case that the formats and path are identical. + if as_bytes(path).startswith(as_bytes(prefix)): + return True + + # attempt to convert the prefix and path both to utf8 + path_utf8 = encodeWithUTF8(path) + prefix_utf8 = encodeWithUTF8(prefix) + if gitConfigBool("core.ignorecase"): - return path.lower().startswith(prefix.lower()) - return path.startswith(prefix) + # Check if we match byte-per-byte. + + return path_utf8.lower().startswith(prefix_utf8.lower()) + return path_utf8.startswith(prefix_utf8) def getClientSpec(): """Look at the p4 client spec, create a View() object that contains @@ -1235,18 +1282,24 @@ def wildcard_decode(path): # Cannot have * in a filename in windows; untested as to # what p4 would do in such a case. if not platform.system() == "Windows": - path = path.replace("%2A", "*") - path = path.replace("%23", "#") \ - .replace("%40", "@") \ - .replace("%25", "%") + path = path.replace(b"%2A", b"*") + path = path.replace(b"%23", b"#") \ + .replace(b"%40", b"@") \ + .replace(b"%25", b"%") return path def wildcard_encode(path): # do % first to avoid double-encoding the %s introduced here - path = path.replace("%", "%25") \ - .replace("*", "%2A") \ - .replace("#", "%23") \ - .replace("@", "%40") + if isinstance(path, unicode): + path = path.replace("%", "%25") \ + .replace("*", "%2A") \ + .replace("#", "%23") \ + .replace("@", "%40") + else: + path = path.replace(b"%", b"%25") \ + .replace(b"*", b"%2A") \ + .replace(b"#", b"%23") \ + .replace(b"@", b"%40") return path def wildcard_present(path): @@ -1378,7 +1431,7 @@ def generatePointer(self, contentFile): ['git', 'lfs', 'pointer', '--file=' + contentFile], stdout=subprocess.PIPE ) - pointerFile = pointerProcess.stdout.read() + pointerFile = as_string(pointerProcess.stdout.read()) if pointerProcess.wait(): os.remove(contentFile) die('git-lfs pointer command failed. Did you install the extension?') @@ -1485,6 +1538,8 @@ def getUserCacheFilename(self): return os.path.join(home, ".gitp4-usercache.txt") def getUserMapFromPerforceServer(self): + """ Creates the usercache from the data in P4. + """ if self.userMapFromPerforceServer: return self.users = {} @@ -1510,18 +1565,22 @@ def getUserMapFromPerforceServer(self): for (key, val) in list(self.users.items()): s += "%s\t%s\n" % (key.expandtabs(1), val.expandtabs(1)) - open(self.getUserCacheFilename(), "wb").write(s) + cache = io.open(self.getUserCacheFilename(), "wb") + cache.write(as_bytes(s)) + cache.close() self.userMapFromPerforceServer = True def loadUserMapFromCache(self): + """ Reads the P4 username to git email map + """ self.users = {} self.userMapFromPerforceServer = False try: - cache = open(self.getUserCacheFilename(), "rb") + cache = io.open(self.getUserCacheFilename(), "rb") lines = cache.readlines() cache.close() for line in lines: - entry = line.strip().split("\t") + entry = as_string(line).strip().split("\t") self.users[entry[0]] = entry[1] except IOError: self.getUserMapFromPerforceServer() @@ -1721,21 +1780,27 @@ def prepareLogMessage(self, template, message, jobs): return result def patchRCSKeywords(self, file, pattern): - # Attempt to zap the RCS keywords in a p4 controlled file matching the given pattern + """ Attempt to zap the RCS keywords in a p4 + controlled file matching the given pattern + """ + bSubLine = as_bytes(r'$\1$') (handle, outFileName) = tempfile.mkstemp(dir='.') try: - outFile = os.fdopen(handle, "w+") - inFile = open(file, "r") - regexp = re.compile(pattern, re.VERBOSE) + outFile = os.fdopen(handle, "w+b") + inFile = open(file, "rb") + regexp = re.compile(as_bytes(pattern), re.VERBOSE) for line in inFile.readlines(): - line = regexp.sub(r'$\1$', line) + line = regexp.sub(bSubLine, line) outFile.write(line) inFile.close() outFile.close() + outFile = None # Forcibly overwrite the original file os.unlink(file) shutil.move(outFileName, file) except: + if outFile != None: + outFile.close() # cleanup our temporary file os.unlink(outFileName) print("Failed to strip RCS keywords in %s" % file) @@ -2139,7 +2204,7 @@ def applyCommit(self, id): tmpFile = os.fdopen(handle, "w+b") if self.isWindows: submitTemplate = submitTemplate.replace("\n", "\r\n") - tmpFile.write(submitTemplate) + tmpFile.write(as_bytes(submitTemplate)) tmpFile.close() if self.prepare_p4_only: @@ -2189,8 +2254,8 @@ def applyCommit(self, id): message = tmpFile.read() tmpFile.close() if self.isWindows: - message = message.replace("\r\n", "\n") - submitTemplate = message[:message.index(separatorLine)] + message = message.replace(b"\r\n", b"\n") + submitTemplate = message[:message.index(as_bytes(separatorLine))] if update_shelve: p4_write_pipe(['shelve', '-r', '-i'], submitTemplate) @@ -2833,8 +2898,11 @@ def stripRepoPath(self, path, prefixes): return path def splitFilesIntoBranches(self, commit): - """Look at each depotFile in the commit to figure out to what - branch it belongs.""" + """ Look at each depotFile in the commit to figure out to what + branch it belongs. + + Data in the commit will NOT be encoded + """ if self.clientSpecDirs: files = self.extractFilesFromCommit(commit) @@ -2875,16 +2943,22 @@ def splitFilesIntoBranches(self, commit): return branches def writeToGitStream(self, gitMode, relPath, contents): - self.gitStream.write('M %s inline %s\n' % (gitMode, relPath)) + """ Writes the bytes[] 'contents' to the git fast-import + with the given 'gitMode' and 'relPath' as the relative + path. + """ + self.gitStream.write('M %s inline %s\n' % (gitMode, as_string(relPath))) self.gitStream.write('data %d\n' % sum(len(d) for d in contents)) for d in contents: - self.gitStream.write(d) + self.gitStreamBytes.write(d) self.gitStream.write('\n') - # output one file from the P4 stream - # - helper for streamP4Files - def streamOneP4File(self, file, contents): + """ output one file from the P4 stream to the git inbound stream. + helper for streamP4files. + + contents should be a bytes (bytes) + """ relPath = self.stripRepoPath(file['depotFile'], self.branchPrefixes) relPath = encodeWithUTF8(relPath, self.verbose) if verbose: @@ -2892,7 +2966,7 @@ def streamOneP4File(self, file, contents): size = int(self.stream_file['fileSize']) else: size = 0 # deleted files don't get a fileSize apparently - sys.stdout.write('\r%s --> %s (%i MB)\n' % (file['depotFile'], relPath, size//1024//1024)) + sys.stdout.write('\r%s --> %s (%i MB)\n' % (path_as_string(file['depotFile']), as_string(relPath), size//1024//1024)) sys.stdout.flush() (type_base, type_mods) = split_p4_type(file["type"]) @@ -2910,7 +2984,7 @@ def streamOneP4File(self, file, contents): # to nothing. This causes p4 errors when checking out such # a change, and errors here too. Work around it by ignoring # the bad symlink; hopefully a future change fixes it. - print("\nIgnoring empty symlink in %s" % file['depotFile']) + print("\nIgnoring empty symlink in %s" % path_as_string(file['depotFile'])) return elif data[-1] == '\n': contents = [data[:-1]] @@ -2950,16 +3024,16 @@ def streamOneP4File(self, file, contents): # Ideally, someday, this script can learn how to generate # appledouble files directly and import those to git, but # non-mac machines can never find a use for apple filetype. - print("\nIgnoring apple filetype file %s" % file['depotFile']) + print("\nIgnoring apple filetype file %s" % path_as_string(file['depotFile'])) return # Note that we do not try to de-mangle keywords on utf16 files, # even though in theory somebody may want that. - pattern = p4_keywords_regexp_for_type(type_base, type_mods) + pattern = as_bytes(p4_keywords_regexp_for_type(type_base, type_mods)) if pattern: regexp = re.compile(pattern, re.VERBOSE) - text = ''.join(contents) - text = regexp.sub(r'$\1$', text) + text = b''.join(contents) + text = regexp.sub(as_bytes(r'$\1$'), text) contents = [ text ] if self.largeFileSystem: @@ -2978,15 +3052,19 @@ def streamOneP4Deletion(self, file): if self.largeFileSystem and self.largeFileSystem.isLargeFile(relPath): self.largeFileSystem.removeLargeFile(relPath) - # handle another chunk of streaming data def streamP4FilesCb(self, marshalled): + """ Callback function for recording P4 chunks of data for streaming + into GIT. + + marshalled data is bytes[] from the caller + """ # catch p4 errors and complain err = None - if "code" in marshalled: - if marshalled["code"] == "error": - if "data" in marshalled: - err = marshalled["data"].rstrip() + if b"code" in marshalled: + if marshalled[b"code"] == b"error": + if b"data" in marshalled: + err = marshalled[b"data"].rstrip() if not err and 'fileSize' in self.stream_file: required_bytes = int((4 * int(self.stream_file["fileSize"])) - calcDiskFree()) @@ -3008,11 +3086,11 @@ def streamP4FilesCb(self, marshalled): # ignore errors, but make sure it exits first self.importProcess.wait() if f: - die("Error from p4 print for %s: %s" % (f, err)) + die("Error from p4 print for %s: %s" % (path_as_string(f), err)) else: die("Error from p4 print: %s" % err) - if 'depotFile' in marshalled and self.stream_have_file_info: + if b'depotFile' in marshalled and self.stream_have_file_info: # start of a new file - output the old one first self.streamOneP4File(self.stream_file, self.stream_contents) self.stream_file = {} @@ -3022,13 +3100,16 @@ def streamP4FilesCb(self, marshalled): # pick up the new file information... for the # 'data' field we need to append to our array for k in list(marshalled.keys()): - if k == 'data': + if k == b'data': if 'streamContentSize' not in self.stream_file: self.stream_file['streamContentSize'] = 0 - self.stream_file['streamContentSize'] += len(marshalled['data']) - self.stream_contents.append(marshalled['data']) + self.stream_file['streamContentSize'] += len(marshalled[b'data']) + self.stream_contents.append(marshalled[b'data']) else: - self.stream_file[k] = marshalled[k] + if k == b'depotFile': + self.stream_file[as_string(k)] = marshalled[k] + else: + self.stream_file[as_string(k)] = as_string(marshalled[k]) if (verbose and 'streamContentSize' in self.stream_file and @@ -3037,13 +3118,14 @@ def streamP4FilesCb(self, marshalled): size = int(self.stream_file["fileSize"]) if size > 0: progress = 100.0*self.stream_file['streamContentSize']/size - sys.stdout.write('\r%s %4.1f%% (%i MB)' % (self.stream_file['depotFile'], progress, int(size//1024//1024))) + sys.stdout.write('\r%s %4.1f%% (%i MB)' % (path_as_string(self.stream_file['depotFile']), progress, int(size//1024//1024))) sys.stdout.flush() self.stream_have_file_info = True - # Stream directly from "p4 files" into "git fast-import" def streamP4Files(self, files): + """ Stream directly from "p4 files" into "git fast-import" + """ filesForCommit = [] filesToRead = [] filesToDelete = [] @@ -3064,7 +3146,7 @@ def streamP4Files(self, files): self.stream_contents = [] self.stream_have_file_info = False - # curry self argument + # Callback for P4 command to collect file content def streamP4FilesCbSelf(entry): self.streamP4FilesCb(entry) @@ -3073,9 +3155,9 @@ def streamP4FilesCbSelf(entry): if 'shelved_cl' in f: # Handle shelved CLs using the "p4 print file@=N" syntax to print # the contents - fileArg = '%s@=%d' % (f['path'], f['shelved_cl']) + fileArg = b'%s@=%d' % (f['path'], as_bytes(f['shelved_cl'])) else: - fileArg = '%s#%s' % (f['path'], f['rev']) + fileArg = b'%s#%s' % (f['path'], as_bytes(f['rev'])) fileArgs.append(fileArg) @@ -3095,7 +3177,7 @@ def make_email(self, userid): def streamTag(self, gitStream, labelName, labelDetails, commit, epoch): """ Stream a p4 tag. - commit is either a git commit, or a fast-import mark, ":" + commit is either a git commit, or a fast-import mark, ":" """ if verbose: @@ -3167,7 +3249,22 @@ def commit(self, details, files, branch, parent = "", allow_empty=False): .format(details['change'])) return + # fast-import: + #'commit' SP LF + #mark? + #original-oid? + #('author' (SP )? SP LT GT SP LF)? + #'committer' (SP )? SP LT GT SP LF + #('encoding' SP )? + #data + #('from' SP LF)? + #('merge' SP LF)* + #(filemodify | filedelete | filecopy | filerename | filedeleteall | notemodify)* + #LF? + + #'commit' - is the name of the branch to make the commit on self.gitStream.write("commit %s\n" % branch) + #'mark' SP : self.gitStream.write("mark :%s\n" % details["change"]) self.committedChanges.add(int(details["change"])) committer = "" @@ -3177,19 +3274,29 @@ def commit(self, details, files, branch, parent = "", allow_empty=False): self.gitStream.write("committer %s\n" % committer) - self.gitStream.write("data < 0: - self.gitStream.write("\nJobs: %s" % (' '.join(jobs))) - + commitText += "\nJobs: %s" % (' '.join(jobs)) if not self.suppress_meta_comment: - self.gitStream.write("\n[git-p4: depot-paths = \"%s\": change = %s" % - (','.join(self.branchPrefixes), details["change"])) - if len(details['options']) > 0: - self.gitStream.write(": options = %s" % details['options']) - self.gitStream.write("]\n") + # coherce the path to the correct formatting in the branch prefixes as well. + dispPaths = [] + for p in self.branchPrefixes: + dispPaths += [path_as_string(p)] - self.gitStream.write("EOT\n\n") + commitText += ("\n[git-p4: depot-paths = \"%s\": change = %s" % + (','.join(dispPaths), details["change"])) + if len(details['options']) > 0: + commitText += (": options = %s" % details['options']) + commitText += "]" + commitText += "\n" + self.gitStream.write("data %s\n" % len(as_bytes(commitText))) + self.gitStream.write(commitText) + self.gitStream.write("\n") if len(parent) > 0: if self.verbose: @@ -3596,30 +3703,35 @@ def sync_origin_only(self): system("git fetch origin") def importHeadRevision(self, revision): - print("Doing initial import of %s from revision %s into %s" % (' '.join(self.depotPaths), revision, self.branch)) - + # Re-encode depot text + dispPaths = [] + utf8Paths = [] + for p in self.depotPaths: + dispPaths += [path_as_string(p)] + print("Doing initial import of %s from revision %s into %s" % (' '.join(dispPaths), revision, self.branch)) details = {} details["user"] = "git perforce import user" - details["desc"] = ("Initial import of %s from the state at revision %s\n" - % (' '.join(self.depotPaths), revision)) + details["desc"] = ("Initial import of %s from the state at revision %s\n" % + (' '.join(dispPaths), revision)) details["change"] = revision newestRevision = 0 + del dispPaths fileCnt = 0 fileArgs = ["%s...%s" % (p,revision) for p in self.depotPaths] - for info in p4CmdList(["files"] + fileArgs): + for info in p4CmdList(["files"] + fileArgs, encode_cmd_output=False): - if 'code' in info and info['code'] == 'error': + if 'code' in info and info['code'] == b'error': sys.stderr.write("p4 returned an error: %s\n" - % info['data']) - if info['data'].find("must refer to client") >= 0: + % as_string(info['data'])) + if info['data'].find(b"must refer to client") >= 0: sys.stderr.write("This particular p4 error is misleading.\n") sys.stderr.write("Perhaps the depot path was misspelled.\n"); sys.stderr.write("Depot path: %s\n" % " ".join(self.depotPaths)) sys.exit(1) if 'p4ExitCode' in info: - sys.stderr.write("p4 exitcode: %s\n" % info['p4ExitCode']) + sys.stderr.write("p4 exitcode: %s\n" % as_string(info['p4ExitCode'])) sys.exit(1) @@ -3632,8 +3744,10 @@ def importHeadRevision(self, revision): #fileCnt = fileCnt + 1 continue + # Save all the file information, howerver do not translate the depotFile name at + # this time. Leave that as bytes since the encoding may vary. for prop in ["depotFile", "rev", "action", "type" ]: - details["%s%s" % (prop, fileCnt)] = info[prop] + details["%s%s" % (prop, fileCnt)] = (info[prop] if prop == "depotFile" else as_string(info[prop])) fileCnt = fileCnt + 1 @@ -3653,13 +3767,18 @@ def importHeadRevision(self, revision): print(self.gitError.read()) def openStreams(self): + """ Opens the fast import pipes. Note that the git* streams are wrapped + to expect Unicode text. To send a raw byte Array, use the importProcess + underlying port + """ self.importProcess = subprocess.Popen(["git", "fast-import"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE); - self.gitOutput = self.importProcess.stdout - self.gitStream = self.importProcess.stdin - self.gitError = self.importProcess.stderr + self.gitOutput = Py23File(self.importProcess.stdout, verbose = self.verbose) + self.gitStream = Py23File(self.importProcess.stdin, verbose = self.verbose) + self.gitError = Py23File(self.importProcess.stderr, verbose = self.verbose) + self.gitStreamBytes = self.importProcess.stdin def closeStreams(self): self.gitStream.close() @@ -4025,15 +4144,17 @@ def run(self, args): self.cloneDestination = depotPaths[-1] depotPaths = depotPaths[:-1] + dispPaths = [] for p in depotPaths: if not p.startswith("//"): sys.stderr.write('Depot paths must start with "//": %s\n' % p) return False + dispPaths += [path_as_string(p)] if not self.cloneDestination: self.cloneDestination = self.defaultDestination(args) - print("Importing from %s into %s" % (', '.join(depotPaths), self.cloneDestination)) + print("Importing from %s into %s" % (', '.join(dispPaths), path_as_string(self.cloneDestination))) if not os.path.exists(self.cloneDestination): os.makedirs(self.cloneDestination) From patchwork Sat Dec 7 17:47:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277715 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E711F139A for ; Sat, 7 Dec 2019 17:48:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B1C30217BA for ; Sat, 7 Dec 2019 17:48:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EGdhyFV7" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726677AbfLGRsC (ORCPT ); Sat, 7 Dec 2019 12:48:02 -0500 Received: from mail-wr1-f65.google.com ([209.85.221.65]:47080 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726637AbfLGRr7 (ORCPT ); Sat, 7 Dec 2019 12:47:59 -0500 Received: by mail-wr1-f65.google.com with SMTP id z7so11228543wrl.13 for ; Sat, 07 Dec 2019 09:47:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=vzEn/mclDF9DnpF7GdPS8pUStnMHWy4LO0bG9HORHwM=; b=EGdhyFV7r758cFgoXisL2T6o5jLMGaaPThkAtSGGZ/niebmJ660XuE/9r/Eh1M4zYu de5HBwO7uRE9anf8L212ly6xjUsMfTPlGWvSggKkzejYfA3UmrpDV5PhjiVGSaPJyif7 e1EsVT/yGf3zTl0/3sY6tF9IHtTvG/cEZJdgHWrz+1Z2ldiT9RGDvd/yo9Y2LENvyL9I lqDIH7/Hmc3NydYrlswjuE4nzWCtU32RSS0yCJiKCKqHMgxO3w+GiEsRtvAIb4HEY42s D2yF54bSBq5SnjkX1SLAWCA7PLqzfrfaY77J+5cwxV4y9uRVXgd3REhA3R3pMAAxf+PM 5m0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=vzEn/mclDF9DnpF7GdPS8pUStnMHWy4LO0bG9HORHwM=; b=jfH7Mck34GqKgnWvZla/RAaCeRT9W4irunIlVMHNEuq2anBAJ2OviMAPVkqH+TWULX GOUJSCDaNYAsxMlCByJAIfEMRIEw68WDT44quWqSbdwNvjTwWlqfR9RPog07MNzATU3M GDxxqvUlIWZABf/WpQlHKgwerLcHErfvsx8JPPZcEklSocVIMKLvjifZfsntUdy8EyKg XrfAcbnKP6J9IhPbEneqqlytHEBS7sNBaiVMsETRYCrT0ccjp3iCNB+SHS4mWpOfVm9d 4OgPa8xxY2QxKPEdp4tgcU9kjKm7OcnkSzytR/8cNHPaD/FpA+eBCV0lB7i82wojsvM6 vlMg== X-Gm-Message-State: APjAAAU0Fw6kPnJbHYtjwupVCxrrBoMW8M6kpIJ4j4Da25xfE82rt/sd kEhyvT+4eG3/Kd7Fo14aTw6hGAfR X-Google-Smtp-Source: APXvYqwtrX7y/4GbxhH34sXXQSKKKUGkQdCaV4fH0E/WTG2P1+wHknmohlgOWTFkI21E0EP15K3efQ== X-Received: by 2002:adf:f803:: with SMTP id s3mr21475388wrp.7.1575740875427; Sat, 07 Dec 2019 09:47:55 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d186sm7748019wmf.7.2019.12.07.09.47.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:55 -0800 (PST) Message-Id: <25ad3e23a337b53ef6ca52019899838cc7ec43f7.1575740863.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:42 +0000 Subject: [PATCH v5 14/15] git-p4: added --encoding parameter to p4 clone Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene The test t9822 did not have any tests that had encoded a directory name in ISO8859-1. Additionally, to make it easier for the user to clone new repositories with a non-UTF-8 encoded path in P4, add a new parameter to p4clone "--encoding" that sets the Add new tests that use ISO8859-1 encoded text in both the directory and file names. git-p4.pathEncoding. Update the View class in the git-p4 code to properly cast text as_string() except for depot path and filenames. Update the documentation to include the new command line parameter for p4clone Signed-off-by: Ben Keene --- Documentation/git-p4.txt | 5 ++ git-p4.py | 57 +++++++++++++----- t/t9822-git-p4-path-encoding.sh | 101 ++++++++++++++++++++++++++++++++ 3 files changed, 147 insertions(+), 16 deletions(-) diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt index 3494a1db3e..8fb844fc49 100644 --- a/Documentation/git-p4.txt +++ b/Documentation/git-p4.txt @@ -305,6 +305,11 @@ options described above. --bare:: Perform a bare clone. See linkgit:git-clone[1]. +--encoding :: + Optionally sets the git-p4.pathEncoding configuration value in + the newly created Git repository before files are synchronized + from P4. See git-p4.pathEncoding for more information. + Submit options ~~~~~~~~~~~~~~ These options can be used to modify 'git p4 submit' behavior. diff --git a/git-p4.py b/git-p4.py index 9cf4e94e28..16f29aae41 100755 --- a/git-p4.py +++ b/git-p4.py @@ -1241,7 +1241,7 @@ def getClientSpec(): entry = specList[0] # the //client/ name - client_name = entry["Client"] + client_name = as_string(entry["Client"]) # just the keys that start with "View" view_keys = [ k for k in list(entry.keys()) if k.startswith("View") ] @@ -2625,19 +2625,25 @@ def run(self, args): return True class View(object): - """Represent a p4 view ("p4 help views"), and map files in a - repo according to the view.""" + """ Represent a p4 view ("p4 help views"), and map files in a + repo according to the view. + """ def __init__(self, client_name): self.mappings = [] - self.client_prefix = "//%s/" % client_name + # the client prefix is saved in bytes as it is used for comparison + # against server data. + self.client_prefix = as_bytes("//%s/" % client_name) # cache results of "p4 where" to lookup client file locations self.client_spec_path_cache = {} def append(self, view_line): - """Parse a view line, splitting it into depot and client - sides. Append to self.mappings, preserving order. This - is only needed for tag creation.""" + """ Parse a view line, splitting it into depot and client + sides. Append to self.mappings, preserving order. This + is only needed for tag creation. + + view_line should be in bytes (depot path encoding) + """ # Split the view line into exactly two words. P4 enforces # structure on these lines that simplifies this quite a bit. @@ -2650,28 +2656,28 @@ def append(self, view_line): # The line is already white-space stripped. # The two words are separated by a single space. # - if view_line[0] == '"': + if view_line[0] == b'"': # First word is double quoted. Find its end. - close_quote_index = view_line.find('"', 1) + close_quote_index = view_line.find(b'"', 1) if close_quote_index <= 0: - die("No first-word closing quote found: %s" % view_line) + die("No first-word closing quote found: %s" % path_as_string(view_line)) depot_side = view_line[1:close_quote_index] # skip closing quote and space rhs_index = close_quote_index + 1 + 1 else: - space_index = view_line.find(" ") + space_index = view_line.find(b" ") if space_index <= 0: - die("No word-splitting space found: %s" % view_line) + die("No word-splitting space found: %s" % path_as_string(view_line)) depot_side = view_line[0:space_index] rhs_index = space_index + 1 # prefix + means overlay on previous mapping - if depot_side.startswith("+"): + if depot_side.startswith(b"+"): depot_side = depot_side[1:] # prefix - means exclude this path, leave out of mappings exclude = False - if depot_side.startswith("-"): + if depot_side.startswith(b"-"): exclude = True depot_side = depot_side[1:] @@ -2682,7 +2688,7 @@ def convert_client_path(self, clientFile): # chop off //client/ part to make it relative if not clientFile.startswith(self.client_prefix): die("No prefix '%s' on clientFile '%s'" % - (self.client_prefix, clientFile)) + (as_string(self.client_prefix)), path_as_string(clientFile)) return clientFile[len(self.client_prefix):] def update_client_spec_path_cache(self, files): @@ -2696,7 +2702,7 @@ def update_client_spec_path_cache(self, files): where_result = p4CmdList(["-x", "-", "where"], stdin=fileArgs, encode_cmd_output=False) for res in where_result: - if "code" in res and res["code"] == "error": + if "code" in res and res["code"] == b"error": # assume error is "... file(s) not in client view" continue if "clientFile" not in res: @@ -4113,10 +4119,14 @@ def __init__(self): help="where to leave result of the clone"), optparse.make_option("--bare", dest="cloneBare", action="store_true", default=False), + optparse.make_option("--encoding", dest="setPathEncoding", + action="store", default=None, + help="Sets the path encoding for this depot") ] self.cloneDestination = None self.needsGit = False self.cloneBare = False + self.setPathEncoding = None def defaultDestination(self, args): """ Returns the last path component as the default git @@ -4140,6 +4150,14 @@ def run(self, args): depotPaths = args + # If we have an encoding provided, ignore what may already exist + # in the registry. This will ensure we show the displayed values + # using the correct encoding. + if self.setPathEncoding: + gitConfigSet("git-p4.pathEncoding", self.setPathEncoding) + + # If more than 1 path element is supplied, the last element + # is the clone destination. if not self.cloneDestination and len(depotPaths) > 1: self.cloneDestination = depotPaths[-1] depotPaths = depotPaths[:-1] @@ -4167,6 +4185,13 @@ def run(self, args): if retcode: raise CalledProcessError(retcode, init_cmd) + # Set the encoding if it was provided command line + if self.setPathEncoding: + init_cmd= ["git", "config", "git-p4.pathEncoding", self.setPathEncoding] + retcode = subprocess.call(init_cmd) + if retcode: + raise CalledProcessError(retcode, init_cmd) + if not P4Sync.run(self, depotPaths): return False diff --git a/t/t9822-git-p4-path-encoding.sh b/t/t9822-git-p4-path-encoding.sh index 572d395498..8d3fe6c5d1 100755 --- a/t/t9822-git-p4-path-encoding.sh +++ b/t/t9822-git-p4-path-encoding.sh @@ -4,9 +4,20 @@ test_description='Clone repositories with non ASCII paths' . ./lib-git-p4.sh +# lowercase filename +# UTF8 - HEX: a-\xc3\xa4_o-\xc3\xb6_u-\xc3\xbc +# - octal: a-\303\244_o-\303\266_u-\303\274 +# ISO8859 - HEX: a-\xe4_o-\xf6_u-\xfc UTF8_ESCAPED="a-\303\244_o-\303\266_u-\303\274.txt" ISO8859_ESCAPED="a-\344_o-\366_u-\374.txt" +# lowercase directory +# UTF8 - HEX: dir_a-\xc3\xa4_o-\xc3\xb6_u-\xc3\xbc +# ISO8859 - HEX: dir_a-\xe4_o-\xf6_u-\xfc +DIR_UTF8_ESCAPED="dir_a-\303\244_o-\303\266_u-\303\274" +DIR_ISO8859_ESCAPED="dir_a-\344_o-\366_u-\374" + + ISO8859="$(printf "$ISO8859_ESCAPED")" && echo content123 >"$ISO8859" && rm "$ISO8859" || { @@ -58,6 +69,22 @@ test_expect_success 'Clone repo containing iso8859-1 encoded paths with git-p4.p ) ' +test_expect_success 'Clone repo containing iso8859-1 encoded paths with using --encoding parameter' ' + test_when_finished cleanup_git && + ( + git p4 clone --encoding iso8859 --destination="$git" //depot && + cd "$git" && + UTF8="$(printf "$UTF8_ESCAPED")" && + echo "$UTF8" >expect && + git -c core.quotepath=false ls-files >actual && + test_cmp expect actual && + + echo content123 >expect && + cat "$UTF8" >actual && + test_cmp expect actual + ) +' + test_expect_success 'Delete iso8859-1 encoded paths and clone' ' ( cd "$cli" && @@ -74,4 +101,78 @@ test_expect_success 'Delete iso8859-1 encoded paths and clone' ' ) ' +# These tests will create a directory with ISO8859-1 characters in both the +# directory and the path. Since it is possible to clone a path instead of using +# the whole client-spec. Check both versions: client-spec and with a direct +# path using --encoding +test_expect_success 'Create a repo containing iso8859-1 encoded directory and filename' ' + ( + DIR_ISO8859="$(printf "$DIR_ISO8859_ESCAPED")" && + ISO8859="$(printf "$ISO8859_ESCAPED")" && + cd "$cli" && + mkdir "$DIR_ISO8859" && + cd "$DIR_ISO8859" && + echo content123 >"$ISO8859" && + p4 add "$ISO8859" && + p4 submit -d "test commit (encoded directory)" + ) +' + +test_expect_success 'Clone repo containing iso8859-1 encoded depot path and files with git-p4.pathEncoding' ' + test_when_finished cleanup_git && + ( + DIR_ISO8859="$(printf "$DIR_ISO8859_ESCAPED")" && + DIR_UTF8="$(printf "$DIR_UTF8_ESCAPED")" && + cd "$git" && + git init . && + git config git-p4.pathEncoding iso8859-1 && + git p4 clone --use-client-spec --destination="$git" "//depot/$DIR_ISO8859" && + cd "$DIR_UTF8" && + UTF8="$(printf "$UTF8_ESCAPED")" && + echo "$UTF8" >expect && + git -c core.quotepath=false ls-files >actual && + test_cmp expect actual && + + echo content123 >expect && + cat "$UTF8" >actual && + test_cmp expect actual + ) +' + +test_expect_success 'Clone repo containing iso8859-1 encoded depot path and files with git-p4.pathEncoding, without --use-client-spec' ' + test_when_finished cleanup_git && + ( + DIR_ISO8859="$(printf "$DIR_ISO8859_ESCAPED")" && + cd "$git" && + git init . && + git config git-p4.pathEncoding iso8859-1 && + git p4 clone --destination="$git" "//depot/$DIR_ISO8859" && + UTF8="$(printf "$UTF8_ESCAPED")" && + echo "$UTF8" >expect && + git -c core.quotepath=false ls-files >actual && + test_cmp expect actual && + + echo content123 >expect && + cat "$UTF8" >actual && + test_cmp expect actual + ) +' + +test_expect_success 'Clone repo containing iso8859-1 encoded depot path and files with using --encoding parameter' ' + test_when_finished cleanup_git && + ( + DIR_ISO8859="$(printf "$DIR_ISO8859_ESCAPED")" && + git p4 clone --encoding iso8859 --destination="$git" "//depot/$DIR_ISO8859" && + cd "$git" && + UTF8="$(printf "$UTF8_ESCAPED")" && + echo "$UTF8" >expect && + git -c core.quotepath=false ls-files >actual && + test_cmp expect actual && + + echo content123 >expect && + cat "$UTF8" >actual && + test_cmp expect actual + ) +' + test_done From patchwork Sat Dec 7 17:47:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 11277713 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8BFE8139A for ; Sat, 7 Dec 2019 17:48:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6A4AF24673 for ; Sat, 7 Dec 2019 17:48:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YBXKL269" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726669AbfLGRsB (ORCPT ); Sat, 7 Dec 2019 12:48:01 -0500 Received: from mail-wr1-f67.google.com ([209.85.221.67]:39074 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726646AbfLGRr6 (ORCPT ); Sat, 7 Dec 2019 12:47:58 -0500 Received: by mail-wr1-f67.google.com with SMTP id y11so11297504wrt.6 for ; Sat, 07 Dec 2019 09:47:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=SLkuqqTjGncvg9DOoCEM8adhCOWvgdXI58i+R9CsCHw=; b=YBXKL269hRPQ8u/yqqFa/jo6POVqCxHIeNVxHupZQ8JRXXWXlmmtEqlWVZT8aL9qIp oC1xM+2agkkbeIwKuZnDKCNtXKnZ25bDEWLWAMAGrYHDUZ9iTsrpm9dZzGGZg7YVVnVM m2PibBlC2xRPrmfWCWbd9jxdacJUinic8x3flyBEHtSBSU3eFgGjUAibg+EiZfWJIMrT Sot8oJshNIPRAINex+OzSrttt5rST1UQ2oiEpvQRvi0/VzFQBg8RA/qk1VZTG3uxvfAC qpO2uLHdpR63k9kTWYNu4IZMHwJa4ueGzdangwFF904RfcXT6/FG9t/ePToJ2ig9W99T WJ1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=SLkuqqTjGncvg9DOoCEM8adhCOWvgdXI58i+R9CsCHw=; b=byjHF4wS2xi0PVgVfpqseleKkQrJrhpv634Ke9ngtPrGnIZM7DG5aJigcJhaHl0Zmb vzcieZKq+TDSACCopslPjF1wib1fgx/Bhw3iw+ZvYL6YLXXMo/tD4mG+/fulEUlTp2LI q4oEGKslE+2jmQRAFUd7eu8xuDGMC55Gn7A1MEriRZT+omZjD+zRGNRshFpgNQ9qRODv N7AqUqhr0FOn9oIHD5ZLo5k/8OxTSIr7OnasL9AQWxmXXfttFdkcKWvOy3oJujvVp8Ne 6FqEoEcMdiSaAKKoGaE814mLnXTfvcv/7T5ZSqNW0gJjADp6QI8QLIe3MylQ4XgYSScI 0vkg== X-Gm-Message-State: APjAAAWpesMivos43/j3EAr+P1OH9OSSvddLd9dCs8NYkcsrnCuy0us3 NpJLc3WWLjI6wbQAVcIo/1+pyN/z X-Google-Smtp-Source: APXvYqxUoyTQwO09XQG7Qk+CBFVdOGa/Qi462rZNxZTfXyxQAo+YnFfmrLnxS02g6VmjfSTFUJ2hBA== X-Received: by 2002:adf:c446:: with SMTP id a6mr21155346wrg.218.1575740876313; Sat, 07 Dec 2019 09:47:56 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p9sm7637839wmg.45.2019.12.07.09.47.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 07 Dec 2019 09:47:55 -0800 (PST) Message-Id: <445dbc59f0cb82fabccc380c0346d65b778d8d1e.1575740863.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Ben Keene via GitGitGadget" Date: Sat, 07 Dec 2019 17:47:43 +0000 Subject: [PATCH v5 15/15] git-p4: Add depot manipulation functions Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ben Keene , Junio C Hamano , Ben Keene Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene Since the Depot paths and filenames are encoded according to P4, we need to track them in bytes but also have to decode them with different encodings (either ASCII or the encoding configured in pathEncoding, which defaults to UTF-8) Add the following functions to support future code conversion actions. * depot_count_depth - counts the number of directories in the path * depot_remove_leading_path - removes (n) directories from the front of the depot path. * depot_Remove_p4_wildcard - removes "/..." from the end of the path * depot_encode_utf8 - converts the path from the native encoding to utf8 encoding. Returns (depot_path, did_decode) * depot_encode_restore - restores the original encoding of the path. Signed-off-by: Ben Keene --- This code block could use review for the depot_encode_* functions. Should this code return an absolute Unicode string or a byte array. --- git-p4.py | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) diff --git a/git-p4.py b/git-p4.py index 16f29aae41..f82f05632c 100755 --- a/git-p4.py +++ b/git-p4.py @@ -724,6 +724,99 @@ def encodeWithUTF8(path, verbose=False): print('Path with non-ASCII characters detected. Used %s to encode: %s ' % (encoding, path)) return path + +def depot_count_depth(depot_path): + """Counts the number of directories found + in the depot_path. Paths will be decoded + with encodeWithUTF8 to ensure that depot + encoding is repected. + + Example: + //depot = 1 + //depot/ = 1 + //depot/dir = 2 + """ + depot_path=encodeWithUTF8(depot_path) + if not depot_path.endswith(b"/"): + depot_path+=b"/" + return depot_path.count(b"/") - 2 + +def depot_remove_leading_path(depot_path, depth): + """Remove depth number of directories from + the beginning of the depot_path. This will + be returned in the original encoding. + The leading "//" does not count as a directory + and will be automatically stripped. + + depot_path should be in bytes + + Example: + Given a depot_path of: //depot/main/file.txt + depth: 0 - depot/main/file.txt + depth: 1 - main/file.txt + depth: 2 - file.txt + depth: 3 - (empty string) + """ + + # First, decode the path + [depot_path, did_decode] = depot_encode_utf8(depot_path) + + #remove leading // + if depot_path.startswith(b"//"): + depot_path=depot_path[2:] + if depth != 0: + segments=depot_path.split(b"/") + segments=segments[depth:] + depot_path=b"/".join(segments) + + if did_decode: + depot_path = depot_encode_restore(depot_path) + + return depot_path + +def depot_remove_p4_wildcard(depot_path): + """Removes the "/..." from the end of depot + path. + + depot_path must be bytes. Bytes are returned. + """ + # First, decode the path + [path, did_decode] = depot_encode_utf8(depot_path) + + if not path.endswith(b"/..."): + return depot_path + path=path[:-4] + + if did_decode: + path = depot_encode_restore(path) + + return path + +def depot_encode_utf8(depot_path): + """conditionally encodes depot_path + in utf8 using the defined pathEncoding. + + Returns a (depot_path, was_encoded)""" + did_decode=False + encoding = 'utf8' + try: + depot_path.decode('ascii', 'strict') + except: + if gitConfig('git-p4.pathEncoding'): + encoding = gitConfig('git-p4.pathEncoding') + depot_path = depot_path.decode(encoding, 'replace').encode('utf8', 'replace') + did_decode=True + return [depot_path, did_decode] + +def depot_encode_restore(encoded_depot_path): + """Recodes an encoded_depot_path + from utf8 back to the configured + pathEncoding""" + encoding = 'utf8' + if gitConfig('git-p4.pathEncoding'): + encoding = gitConfig('git-p4.pathEncoding') + return encoded_depot_path.decode('utf8', 'replace').encode(encoding, 'replace') + class P4Exception(Exception): """ Base class for exceptions from the p4 client """ def __init__(self, exit_code):