[1/1] Travis CI: Lint for Python syntax errors and undefined names

Commit Message

Philippe Blain via GitGitGadget July 19, 2019, 2:18 p.m. UTC

From: cclauss <[e-mail]>

Several things were changed between Python 2 and Python 3.
There are a few Python 3 incompatibilities to work on.
Here we are making changes to make the code run on both Py2 and Py3.
We are doing this because the end of life of Python 2 is in 167 days.
We are using print() function because legacy print statements are syntax
errors on Py3.
reduce() was moved in Python 3 and raw_input() was removed so we make
changes to avoid NameErrors being raised at runtime.
We are also putting flake8 lint tests in place on Travis CI to avoid
any backsliding on future pull requests.

Signed-off-by: cclauss <[e-mail]>
---
 .travis.yml                        |  4 +++
 contrib/fast-import/import-zips.py |  3 +-
 contrib/hg-to-git/hg-to-git.py     | 47 +++++++++++++++---------------
 git-p4.py                          |  3 ++
 4 files changed, 33 insertions(+), 24 deletions(-)

Comments

Junio C Hamano July 19, 2019, 7:44 p.m. UTC | #1

"cclauss via GitGitGadget" <[e-mail]> writes:

> From: cclauss <[e-mail]>

I'll tweak this line (and your sign-off) to read as "Christian
Clauss <[e-mail]>" as you had in your cover letter.

> Several things were changed between Python 2 and Python 3.
> There are a few Python 3 incompatibilities to work on.
> Here we are making changes to make the code run on both Py2 and Py3.
> We are doing this because the end of life of Python 2 is in 167 days.

All sounds sensible, and the above is quite a good problem
description.

> We are using print() function because legacy print statements are syntax
> errors on Py3.
> reduce() was moved in Python 3 and raw_input() was removed so we make
> changes to avoid NameErrors being raised at runtime.
> We are also putting flake8 lint tests in place on Travis CI to avoid
> any backsliding on future pull requests.

Nothing is wrong here, but the convention in our codebase is to
describe the changes as if we are giving orders to the codebase "to
be like so".  And as you have enumeration, I would write something
like this if I were describing this change:


     - Use the `print()` function, because Py3 no longer has the `print`
       statement.

     - Import `reduce()` from functools, because Py3 requires this, and
       importing also works with Py2 (even though it wasn't necessary).

     - Use `input()` instead of `raw_input()`, as the former can be used
       with both but the latter was removed in Py3.

    Also add a CI job to Travis CI to run flake8 lint to avoid an
    backsliding on future pull requests.

https://python-future.org/compatible_idioms.html#raw-input seems to
suggest, just like you import reduce from functools, you need to
import input from builtins.  Is it not the case?

> +      script: flake8 . --count --select=E9,F63,F72,F82 --show-source --statistics

How was the set of "select"ed violations to catch chosen?  How are
we going to maintain this list going forward?

The rest of the patch looked sensible.

Thanks.

SZEDER Gábor July 20, 2019, 12:51 p.m. UTC | #2

On Fri, Jul 19, 2019 at 07:18:55AM -0700, cclauss via GitGitGadget wrote:
> Several things were changed between Python 2 and Python 3.
> There are a few Python 3 incompatibilities to work on.
> Here we are making changes to make the code run on both Py2 and Py3.
> We are doing this because the end of life of Python 2 is in 167 days.
> We are using print() function because legacy print statements are syntax
> errors on Py3.
> reduce() was moved in Python 3 and raw_input() was removed so we make
> changes to avoid NameErrors being raised at runtime.
> We are also putting flake8 lint tests in place on Travis CI to avoid
> any backsliding on future pull requests.

It seems to me that this patch does too many things at once, and
perhaps it would be better to split it into a couple of smaller
patches that do only one thing, e.g.:

  - use print function instead of statement in 'hg-to-git' (which
    constitutes the bulk of this patch),
  - do the same in 'contrib/fast-import/import-zips.py'
  - import 'reduce' and fix 'raw_input' in 'git-p4'
  - and once all that is done and the linter runs clean, add the
    linter job to Travis CI.

This would ease the job of the reader, now and in the future, and it
would better stand out that ...

> diff --git a/git-p4.py b/git-p4.py
> index 3991e7d1a7..9faee25db2 100755
> --- a/git-p4.py
> +++ b/git-p4.py

> @@ -3968,6 +3970,7 @@ def renameBranch(self, branch_name):
>                  break
>  
>          if not found:
> +            sync = P4Sync()

... this change is not mentioned in the commit message.

Is this something the linter complains about?
It doesn't look like a Python 2 vs. 3 compatibility fix to me, so it
might deserve a dedicated patch.

Cc-ing Luke for this bit.

>              sys.exit("gave up trying to rename existing branch {0}".format(sync.branch))
>  
>      def findLastP4Revision(self, starting_point):
> -- 
> gitgitgadget

Patch

diff --git a/.travis.yml b/.travis.yml
index ffb1bc46f2..62557dc8fd 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -39,6 +39,10 @@  matrix:
       compiler:
       script: ci/test-documentation.sh
       after_failure:
+    - env: jobname=LintPython
+      language: python
+      before_install: pip install flake8
+      script: flake8 . --count --select=E9,F63,F72,F82 --show-source --statistics
 
 before_install: ci/install-dependencies.sh
 script: ci/run-build-and-tests.sh
diff --git a/contrib/fast-import/import-zips.py b/contrib/fast-import/import-zips.py
index d12c296223..138194475c 100755
--- a/contrib/fast-import/import-zips.py
+++ b/contrib/fast-import/import-zips.py
@@ -8,6 +8,7 @@ 
 ##  python import-zips.py *.zip
 ##  git log --stat import-zips
 
+from __future__ import print_function
 from os import popen, path
 from sys import argv, exit, hexversion, stderr
 from time import mktime
@@ -19,7 +20,7 @@ 
     exit(1)
 
 if len(argv) < 2:
-    print 'usage:', argv[0], '<zipfile>...'
+    print('usage:', argv[0], '<zipfile>...')
     exit(1)
 
 branch_ref = 'refs/heads/import-zips'
diff --git a/contrib/hg-to-git/hg-to-git.py b/contrib/hg-to-git/hg-to-git.py
index de3f81667e..1d5e5156a4 100755
--- a/contrib/hg-to-git/hg-to-git.py
+++ b/contrib/hg-to-git/hg-to-git.py
@@ -18,6 +18,7 @@ 
     along with this program; if not, see <http://www.gnu.org/licenses/>.
 """
 
+from __future__ import print_function
 import os, os.path, sys
 import tempfile, pickle, getopt
 import re
@@ -42,7 +43,7 @@ 
 
 def usage():
 
-        print """\
+        print("""\
 %s: [OPTIONS] <hgprj>
 
 options:
@@ -54,7 +55,7 @@  def usage():
 
 required:
     hgprj:  name of the HG project to import (directory)
-""" % sys.argv[0]
+""" % sys.argv[0])
 
 #------------------------------------------------------------------------------
 
@@ -104,22 +105,22 @@  def getgitenv(user, date):
 if state:
     if os.path.exists(state):
         if verbose:
-            print 'State does exist, reading'
+            print('State does exist, reading')
         f = open(state, 'r')
         hgvers = pickle.load(f)
     else:
-        print 'State does not exist, first run'
+        print('State does not exist, first run')
 
 sock = os.popen('hg tip --template "{rev}"')
 tip = sock.read()
 if sock.close():
     sys.exit(1)
 if verbose:
-    print 'tip is', tip
+    print('tip is', tip)
 
 # Calculate the branches
 if verbose:
-    print 'analysing the branches...'
+    print('analysing the branches...')
 hgchildren["0"] = ()
 hgparents["0"] = (None, None)
 hgbranch["0"] = "master"
@@ -155,7 +156,7 @@  def getgitenv(user, date):
             hgbranch[str(cset)] = "branch-" + str(cset)
 
 if not hgvers.has_key("0"):
-    print 'creating repository'
+    print('creating repository')
     os.system('git init')
 
 # loop through every hg changeset
@@ -180,27 +181,27 @@  def getgitenv(user, date):
     os.write(fdcomment, csetcomment)
     os.close(fdcomment)
 
-    print '-----------------------------------------'
-    print 'cset:', cset
-    print 'branch:', hgbranch[str(cset)]
-    print 'user:', user
-    print 'date:', date
-    print 'comment:', csetcomment
+    print('-----------------------------------------')
+    print('cset:', cset)
+    print('branch:', hgbranch[str(cset)])
+    print('user:', user)
+    print('date:', date)
+    print('comment:', csetcomment)
     if parent:
-	print 'parent:', parent
+	    print('parent:', parent)
     if mparent:
-        print 'mparent:', mparent
+        print('mparent:', mparent)
     if tag:
-        print 'tag:', tag
-    print '-----------------------------------------'
+        print('tag:', tag)
+    print('-----------------------------------------')
 
     # checkout the parent if necessary
     if cset != 0:
         if hgbranch[str(cset)] == "branch-" + str(cset):
-            print 'creating new branch', hgbranch[str(cset)]
+            print('creating new branch', hgbranch[str(cset)])
             os.system('git checkout -b %s %s' % (hgbranch[str(cset)], hgvers[parent]))
         else:
-            print 'checking out branch', hgbranch[str(cset)]
+            print('checking out branch', hgbranch[str(cset)])
             os.system('git checkout %s' % hgbranch[str(cset)])
 
     # merge
@@ -209,7 +210,7 @@  def getgitenv(user, date):
             otherbranch = hgbranch[mparent]
         else:
             otherbranch = hgbranch[parent]
-        print 'merging', otherbranch, 'into', hgbranch[str(cset)]
+        print('merging', otherbranch, 'into', hgbranch[str(cset)])
         os.system(getgitenv(user, date) + 'git merge --no-commit -s ours "" %s %s' % (hgbranch[str(cset)], otherbranch))
 
     # remove everything except .git and .hg directories
@@ -233,12 +234,12 @@  def getgitenv(user, date):
 
     # delete branch if not used anymore...
     if mparent and len(hgchildren[str(cset)]):
-        print "Deleting unused branch:", otherbranch
+        print("Deleting unused branch:", otherbranch)
         os.system('git branch -d %s' % otherbranch)
 
     # retrieve and record the version
     vvv = os.popen('git show --quiet --pretty=format:%H').read()
-    print 'record', cset, '->', vvv
+    print('record', cset, '->', vvv)
     hgvers[str(cset)] = vvv
 
 if hgnewcsets >= opt_nrepack and opt_nrepack != -1:
@@ -247,7 +248,7 @@  def getgitenv(user, date):
 # write the state for incrementals
 if state:
     if verbose:
-        print 'Writing state'
+        print('Writing state')
     f = open(state, 'w')
     pickle.dump(hgvers, f)
 
diff --git a/git-p4.py b/git-p4.py
index 3991e7d1a7..9faee25db2 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -36,6 +36,8 @@ 
     unicode = str
     bytes = bytes
     basestring = (str,bytes)
+    raw_input = input
+    from functools import reduce
 else:
     # 'unicode' exists, must be Python 2
     str = str
@@ -3968,6 +3970,7 @@  def renameBranch(self, branch_name):
                 break
 
         if not found:
+            sync = P4Sync()
             sys.exit("gave up trying to rename existing branch {0}".format(sync.branch))
 
     def findLastP4Revision(self, starting_point):