[RFC,v5,026/126] python: add commit-per-subsystem.py

Commit Message

Vladimir Sementsov-Ogievskiy Oct. 11, 2019, 4:04 p.m. UTC

Add script to automatically commit tree-wide changes per-subsystem.

Signed-off-by: Vladimir Sementsov-Ogievskiy <[e-mail]>
---

CC: Gerd Hoffmann <[e-mail]>
CC: "Gonglei (Arei)" <[e-mail]>
CC: Eduardo Habkost <[e-mail]>
CC: Igor Mammedov <[e-mail]>
CC: Laurent Vivier <[e-mail]>
CC: Amit Shah <[e-mail]>
CC: Kevin Wolf <[e-mail]>
CC: Max Reitz <[e-mail]>
CC: John Snow <[e-mail]>
CC: Ari Sundholm <[e-mail]>
CC: Pavel Dovgalyuk <[e-mail]>
CC: Paolo Bonzini <[e-mail]>
CC: Stefan Hajnoczi <[e-mail]>
CC: Fam Zheng <[e-mail]>
CC: Stefan Weil <[e-mail]>
CC: Ronnie Sahlberg <[e-mail]>
CC: Peter Lieven <[e-mail]>
CC: Eric Blake <[e-mail]>
CC: "Denis V. Lunev" <[e-mail]>
CC: Markus Armbruster <[e-mail]>
CC: Alberto Garcia <[e-mail]>
CC: Jason Dillaman <[e-mail]>
CC: Wen Congyang <[e-mail]>
CC: Xie Changlong <[e-mail]>
CC: Liu Yuan <[e-mail]>
CC: "Richard W.M. Jones" <[e-mail]>
CC: Jeff Cody <[e-mail]>
CC: "Marc-André Lureau" <[e-mail]>
CC: "Daniel P. Berrangé" <[e-mail]>
CC: Richard Henderson <[e-mail]>
CC: Greg Kurz <[e-mail]>
CC: "Michael S. Tsirkin" <[e-mail]>
CC: Marcel Apfelbaum <[e-mail]>
CC: Beniamino Galvani <[e-mail]>
CC: Peter Maydell <[e-mail]>
CC: "Cédric Le Goater" <[e-mail]>
CC: Andrew Jeffery <[e-mail]>
CC: Joel Stanley <[e-mail]>
CC: Andrew Baumann <[e-mail]>
CC: "Philippe Mathieu-Daudé" <[e-mail]>
CC: Antony Pavlov <[e-mail]>
CC: Jean-Christophe Dubois <[e-mail]>
CC: Peter Chubb <[e-mail]>
CC: Subbaraya Sundeep <[e-mail]>
CC: Eric Auger <[e-mail]>
CC: Alistair Francis <[e-mail]>
CC: "Edgar E. Iglesias" <[e-mail]>
CC: Stefano Stabellini <[e-mail]>
CC: Anthony Perard <[e-mail]>
CC: Paul Durrant <[e-mail]>
CC: Paul Burton <[e-mail]>
CC: Aleksandar Rikalo <[e-mail]>
CC: Chris Wulff <[e-mail]>
CC: Marek Vasut <[e-mail]>
CC: David Gibson <[e-mail]>
CC: Cornelia Huck <[e-mail]>
CC: Halil Pasic <[e-mail]>
CC: Christian Borntraeger <[e-mail]>
CC: "Hervé Poussineau" <[e-mail]>
CC: Xiao Guangrong <[e-mail]>
CC: Aurelien Jarno <[e-mail]>
CC: Aleksandar Markovic <[e-mail]>
CC: Mark Cave-Ayland <[e-mail]>
CC: Jason Wang <[e-mail]>
CC: Laszlo Ersek <[e-mail]>
CC: Yuval Shaia <[e-mail]>
CC: Palmer Dabbelt <[e-mail]>
CC: Sagar Karandikar <[e-mail]>
CC: Bastian Koppelmann <[e-mail]>
CC: David Hildenbrand <[e-mail]>
CC: Thomas Huth <[e-mail]>
CC: Eric Farman <[e-mail]>
CC: Matthew Rosato <[e-mail]>
CC: Hannes Reinecke <[e-mail]>
CC: Michael Walle <[e-mail]>
CC: Artyom Tarasenko <[e-mail]>
CC: Stefan Berger <[e-mail]>
CC: Samuel Thibault <[e-mail]>
CC: Alex Williamson <[e-mail]>
CC: Tony Krowiak <[e-mail]>
CC: Pierre Morel <[e-mail]>
CC: Michael Roth <[e-mail]>
CC: Hailiang Zhang <[e-mail]>
CC: Juan Quintela <[e-mail]>
CC: "Dr. David Alan Gilbert" <[e-mail]>
CC: Luigi Rizzo <[e-mail]>
CC: Giuseppe Lettieri <[e-mail]>
CC: Vincenzo Maffione <[e-mail]>
CC: Jan Kiszka <[e-mail]>
CC: Anthony Green <[e-mail]>
CC: Stafford Horne <[e-mail]>
CC: Guan Xuetao <[e-mail]>
CC: Max Filippov <[e-mail]>
CC: [e-mail]
CC: [e-mail]
CC: [e-mail]
CC: [e-mail]
CC: [e-mail]
CC: [e-mail]
CC: [e-mail]
CC: [e-mail]

 python/commit-per-subsystem.py | 204 +++++++++++++++++++++++++++++++++
 1 file changed, 204 insertions(+)
 create mode 100755 python/commit-per-subsystem.py

Comments

Marc-André Lureau Nov. 8, 2019, 9:18 p.m. UTC | #1

Hi

On Fri, Oct 11, 2019 at 9:11 PM Vladimir Sementsov-Ogievskiy
<[e-mail]> wrote:
>
> Add script to automatically commit tree-wide changes per-subsystem.

Oh interesting! I guess it could use a --help or a larger commit
message to explain a bit what it does (I imagine from the rest of the
series, but someone looking at the script without context may wonder;)

You could also fix some pep8/pylint/pycodestyle

>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <[e-mail]>
> ---
>
> CC: Gerd Hoffmann <[e-mail]>
> CC: "Gonglei (Arei)" <[e-mail]>
> CC: Eduardo Habkost <[e-mail]>
> CC: Igor Mammedov <[e-mail]>
> CC: Laurent Vivier <[e-mail]>
> CC: Amit Shah <[e-mail]>
> CC: Kevin Wolf <[e-mail]>
> CC: Max Reitz <[e-mail]>
> CC: John Snow <[e-mail]>
> CC: Ari Sundholm <[e-mail]>
> CC: Pavel Dovgalyuk <[e-mail]>
> CC: Paolo Bonzini <[e-mail]>
> CC: Stefan Hajnoczi <[e-mail]>
> CC: Fam Zheng <[e-mail]>
> CC: Stefan Weil <[e-mail]>
> CC: Ronnie Sahlberg <[e-mail]>
> CC: Peter Lieven <[e-mail]>
> CC: Eric Blake <[e-mail]>
> CC: "Denis V. Lunev" <[e-mail]>
> CC: Markus Armbruster <[e-mail]>
> CC: Alberto Garcia <[e-mail]>
> CC: Jason Dillaman <[e-mail]>
> CC: Wen Congyang <[e-mail]>
> CC: Xie Changlong <[e-mail]>
> CC: Liu Yuan <[e-mail]>
> CC: "Richard W.M. Jones" <[e-mail]>
> CC: Jeff Cody <[e-mail]>
> CC: "Marc-André Lureau" <[e-mail]>
> CC: "Daniel P. Berrangé" <[e-mail]>
> CC: Richard Henderson <[e-mail]>
> CC: Greg Kurz <[e-mail]>
> CC: "Michael S. Tsirkin" <[e-mail]>
> CC: Marcel Apfelbaum <[e-mail]>
> CC: Beniamino Galvani <[e-mail]>
> CC: Peter Maydell <[e-mail]>
> CC: "Cédric Le Goater" <[e-mail]>
> CC: Andrew Jeffery <[e-mail]>
> CC: Joel Stanley <[e-mail]>
> CC: Andrew Baumann <[e-mail]>
> CC: "Philippe Mathieu-Daudé" <[e-mail]>
> CC: Antony Pavlov <[e-mail]>
> CC: Jean-Christophe Dubois <[e-mail]>
> CC: Peter Chubb <[e-mail]>
> CC: Subbaraya Sundeep <[e-mail]>
> CC: Eric Auger <[e-mail]>
> CC: Alistair Francis <[e-mail]>
> CC: "Edgar E. Iglesias" <[e-mail]>
> CC: Stefano Stabellini <[e-mail]>
> CC: Anthony Perard <[e-mail]>
> CC: Paul Durrant <[e-mail]>
> CC: Paul Burton <[e-mail]>
> CC: Aleksandar Rikalo <[e-mail]>
> CC: Chris Wulff <[e-mail]>
> CC: Marek Vasut <[e-mail]>
> CC: David Gibson <[e-mail]>
> CC: Cornelia Huck <[e-mail]>
> CC: Halil Pasic <[e-mail]>
> CC: Christian Borntraeger <[e-mail]>
> CC: "Hervé Poussineau" <[e-mail]>
> CC: Xiao Guangrong <[e-mail]>
> CC: Aurelien Jarno <[e-mail]>
> CC: Aleksandar Markovic <[e-mail]>
> CC: Mark Cave-Ayland <[e-mail]>
> CC: Jason Wang <[e-mail]>
> CC: Laszlo Ersek <[e-mail]>
> CC: Yuval Shaia <[e-mail]>
> CC: Palmer Dabbelt <[e-mail]>
> CC: Sagar Karandikar <[e-mail]>
> CC: Bastian Koppelmann <[e-mail]>
> CC: David Hildenbrand <[e-mail]>
> CC: Thomas Huth <[e-mail]>
> CC: Eric Farman <[e-mail]>
> CC: Matthew Rosato <[e-mail]>
> CC: Hannes Reinecke <[e-mail]>
> CC: Michael Walle <[e-mail]>
> CC: Artyom Tarasenko <[e-mail]>
> CC: Stefan Berger <[e-mail]>
> CC: Samuel Thibault <[e-mail]>
> CC: Alex Williamson <[e-mail]>
> CC: Tony Krowiak <[e-mail]>
> CC: Pierre Morel <[e-mail]>
> CC: Michael Roth <[e-mail]>
> CC: Hailiang Zhang <[e-mail]>
> CC: Juan Quintela <[e-mail]>
> CC: "Dr. David Alan Gilbert" <[e-mail]>
> CC: Luigi Rizzo <[e-mail]>
> CC: Giuseppe Lettieri <[e-mail]>
> CC: Vincenzo Maffione <[e-mail]>
> CC: Jan Kiszka <[e-mail]>
> CC: Anthony Green <[e-mail]>
> CC: Stafford Horne <[e-mail]>
> CC: Guan Xuetao <[e-mail]>
> CC: Max Filippov <[e-mail]>
> CC: [e-mail]
> CC: [e-mail]
> CC: [e-mail]
> CC: [e-mail]
> CC: [e-mail]
> CC: [e-mail]
> CC: [e-mail]
> CC: [e-mail]
>
>  python/commit-per-subsystem.py | 204 +++++++++++++++++++++++++++++++++
>  1 file changed, 204 insertions(+)
>  create mode 100755 python/commit-per-subsystem.py
>
> diff --git a/python/commit-per-subsystem.py b/python/commit-per-subsystem.py
> new file mode 100755
> index 0000000000..2ccf84cb15
> --- /dev/null
> +++ b/python/commit-per-subsystem.py
> @@ -0,0 +1,204 @@
> +#!/usr/bin/env python3
> +#
> +# Copyright (c) 2019 Virtuozzo International GmbH
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +import subprocess
> +import sys
> +import os
> +import glob
> +
> +
> +def git_add(pattern):
> +    subprocess.run(['git', 'add', pattern])
> +
> +
> +def git_commit(msg):
> +    subprocess.run(['git', 'commit', '-m', msg], capture_output=True)
> +
> +
> +def git_changed_files():
> +    ret = subprocess.check_output(['git', 'diff', '--name-only'], encoding='utf-8').split('\n')
> +    if ret[-1] == '':
> +        del ret[-1]
> +    return ret
> +
> +
> +maintainers = sys.argv[1]
> +message = sys.argv[2].strip()
> +
> +subsystem = None
> +
> +remap = {
> +    'Block layer core': 'block',
> +    'Block Jobs': 'block',
> +    'Dirty Bitmaps': 'block',
> +    'Block QAPI, monitor, command line': 'block',
> +    'Block I/O path': 'block',
> +    'Throttling infrastructure': 'block',
> +    'Architecture support': 's390x',
> +    'Guest CPU Cores (KVM)': 'kvm',
> +    'Guest CPU Cores (Xen)': 'xen',
> +    'Guest CPU cores (TCG)': 'tcg',
> +    'Network Block Device (NBD)': 'nbd',
> +    'Parallel NOR Flash devices': 'pflash',
> +    'Firmware configuration (fw_cfg)': 'fw_cfg',
> +    'Block SCSI subsystem': 'scsi',
> +    'Network device backends': 'net',
> +    'Netmap network backend': 'net',
> +    'Host Memory Backends': 'hostmem',
> +    'Cryptodev Backends': 'cryptodev',
> +    'QEMU Guest Agent': 'qga',
> +    'COLO Framework': 'colo',
> +    'Command line option argument parsing': 'cmdline',
> +    'Character device backends': 'chardev'
> +}
> +
> +
> +class Maintainers:
> +    def add(self, subsystem, path, mapper, mapper_name, glob_count=1):
> +        if subsystem in remap:
> +            subsystem = remap[subsystem]
> +        if subsystem not in self.subsystems:
> +            self.subsystems.append(subsystem)
> +
> +        if path[-1] == '/':
> +            path = path[:-1]
> +
> +        if path in mapper:
> +            if mapper[path][1] == glob_count:
> +                print('Warning: "{}" both in "{}" and "{}" in {} mapper with '
> +                      'same glob-count={}. {} ignored for this path.'.format(
> +                        path, mapper[path][0], subsystem, mapper_name, glob_count,
> +                          subsystem))
> +                return
> +            if mapper[path][1] < glob_count:
> +                # silently ignore worse match
> +                return
> +
> +        mapper[path] = (subsystem, glob_count)
> +
> +    def __init__(self, file_name):
> +        self.map_file = {}
> +        self.map_glob_file = {}
> +        self.map_dir = {}
> +        self.map_glob_dir = {}
> +        self.map_unmaintained_dir = {
> +            'python': ('python', 1),
> +            'hw/misc': ('misc', 1)
> +        }
> +        self.subsystems = ['python', 'misc']
> +        subsystem = None
> +
> +        with open(file_name) as f:
> +            mode2 = False
> +            prevline = ''
> +            for line in f:
> +                line = line.rstrip()
> +                if not line:
> +                    continue
> +                if len(line) >= 2 and line[1] == ':':
> +                    if line[0] == 'F':
> +                        fname = line[3:]
> +                        if fname in ['*', '*/']:
> +                            continue
> +                        if os.path.isfile(fname):
> +                            self.add(subsystem, fname, self.map_file, 'file')
> +                        elif os.path.isdir(fname):
> +                            self.add(subsystem, fname, self.map_dir, 'dir')
> +                        else:
> +                            paths = glob.glob(fname)
> +                            if not paths:
> +                                print('Warning: nothing corresponds to "{}"'.format(fname))
> +                                continue
> +
> +                            n = len(paths)
> +                            for f in paths:
> +                                if os.path.isfile(f):
> +                                    self.add(subsystem, f, self.map_glob_file, 'glob-file', n)
> +                                else:
> +                                    assert os.path.isdir(f)
> +                                    self.add(subsystem, f, self.map_glob_dir, 'glob-dir', n)
> +                elif line[:3] == '---':
> +                    subsystem = prevline
> +                    if subsystem == 'Devices':
> +                        mode2 = True
> +                elif mode2:
> +                    subsystem = line
> +                prevline = line
> +
> +    def find_in_map_dir(self, file_name, mapper):
> +        while file_name != '' and file_name not in mapper:
> +            file_name = os.path.dirname(file_name)
> +
> +        return None if file_name == '' else mapper[file_name][0]
> +
> +    def find_in_map_file(self, file_name, mapper):
> +        if file_name in mapper:
> +            return mapper[file_name][0]
> +
> +    def find_subsystem(self, file_name):
> +        s = self.find_in_map_file(file_name, self.map_file)
> +        if s is not None:
> +            return s
> +
> +        s = self.find_in_map_file(file_name, self.map_glob_file)
> +        if s is not None:
> +            return s
> +
> +        s = self.find_in_map_dir(file_name, self.map_dir)
> +        if s is not None:
> +            return s
> +
> +        s = self.find_in_map_dir(file_name, self.map_glob_dir)
> +        if s is not None:
> +            return s
> +
> +        s = self.find_in_map_dir(file_name, self.map_unmaintained_dir)
> +        if s is not None:
> +            return s
> +
> +        self.subsystems.append(file_name)
> +        return file_name
> +
> +
> +def commit(subsystem):
> +    msg = subsystem
> +    if msg in remap:
> +        msg = remap[msg]
> +    msg += ': ' + message
> +    git_commit(msg)
> +
> +mnt = Maintainers(maintainers)
> +res = {}
> +for f in git_changed_files():
> +    s = mnt.find_subsystem(f)
> +    if s in res:
> +        res[s].append(f)
> +    else:
> +        res[s] = [f]
> +
> +for s in mnt.subsystems:
> +    if s in res:
> +        print(s)
> +        for f in res[s]:
> +            print('  ', f)
> +
> +for s in mnt.subsystems:
> +    if s in res:
> +        for f in res[s]:
> +            git_add(f)
> +        commit(s)
> --
> 2.21.0
>
>

Aleksandar Markovic Nov. 11, 2019, 4:37 p.m. UTC | #2

On Friday, October 11, 2019, Vladimir Sementsov-Ogievskiy <
[e-mail]> wrote:

> Add script to automatically commit tree-wide changes per-subsystem.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <[e-mail]>
> ---


Great idea!

Can you just add a comment somewhere close to the top of the file on script
usage? Or "--help" option? If you would like to be the script maintainer,
please change the MAINTAINERS too.

Reviewed-by: Aleksandar Markovic <[e-mail]>


>
> CC: Gerd Hoffmann <[e-mail]>
> CC: "Gonglei (Arei)" <[e-mail]>
> CC: Eduardo Habkost <[e-mail]>
> CC: Igor Mammedov <[e-mail]>
> CC: Laurent Vivier <[e-mail]>
> CC: Amit Shah <[e-mail]>
> CC: Kevin Wolf <[e-mail]>
> CC: Max Reitz <[e-mail]>
> CC: John Snow <[e-mail]>
> CC: Ari Sundholm <[e-mail]>
> CC: Pavel Dovgalyuk <[e-mail]>
> CC: Paolo Bonzini <[e-mail]>
> CC: Stefan Hajnoczi <[e-mail]>
> CC: Fam Zheng <[e-mail]>
> CC: Stefan Weil <[e-mail]>
> CC: Ronnie Sahlberg <[e-mail]>
> CC: Peter Lieven <[e-mail]>
> CC: Eric Blake <[e-mail]>
> CC: "Denis V. Lunev" <[e-mail]>
> CC: Markus Armbruster <[e-mail]>
> CC: Alberto Garcia <[e-mail]>
> CC: Jason Dillaman <[e-mail]>
> CC: Wen Congyang <[e-mail]>
> CC: Xie Changlong <[e-mail]>
> CC: Liu Yuan <[e-mail]>
> CC: "Richard W.M. Jones" <[e-mail]>
> CC: Jeff Cody <[e-mail]>
> CC: "Marc-André Lureau" <[e-mail]>
> CC: "Daniel P. Berrangé" <[e-mail]>
> CC: Richard Henderson <[e-mail]>
> CC: Greg Kurz <[e-mail]>
> CC: "Michael S. Tsirkin" <[e-mail]>
> CC: Marcel Apfelbaum <[e-mail]>
> CC: Beniamino Galvani <[e-mail]>
> CC: Peter Maydell <[e-mail]>
> CC: "Cédric Le Goater" <[e-mail]>
> CC: Andrew Jeffery <[e-mail]>
> CC: Joel Stanley <[e-mail]>
> CC: Andrew Baumann <[e-mail]>
> CC: "Philippe Mathieu-Daudé" <[e-mail]>
> CC: Antony Pavlov <[e-mail]>
> CC: Jean-Christophe Dubois <[e-mail]>
> CC: Peter Chubb <[e-mail]>
> CC: Subbaraya Sundeep <[e-mail]>
> CC: Eric Auger <[e-mail]>
> CC: Alistair Francis <[e-mail]>
> CC: "Edgar E. Iglesias" <[e-mail]>
> CC: Stefano Stabellini <[e-mail]>
> CC: Anthony Perard <[e-mail]>
> CC: Paul Durrant <[e-mail]>
> CC: Paul Burton <[e-mail]>
> CC: Aleksandar Rikalo <[e-mail]>
> CC: Chris Wulff <[e-mail]>
> CC: Marek Vasut <[e-mail]>
> CC: David Gibson <[e-mail]>
> CC: Cornelia Huck <[e-mail]>
> CC: Halil Pasic <[e-mail]>
> CC: Christian Borntraeger <[e-mail]>
> CC: "Hervé Poussineau" <[e-mail]>
> CC: Xiao Guangrong <[e-mail]>
> CC: Aurelien Jarno <[e-mail]>
> CC: Aleksandar Markovic <[e-mail]>
> CC: Mark Cave-Ayland <[e-mail]>
> CC: Jason Wang <[e-mail]>
> CC: Laszlo Ersek <[e-mail]>
> CC: Yuval Shaia <[e-mail]>
> CC: Palmer Dabbelt <[e-mail]>
> CC: Sagar Karandikar <[e-mail]>
> CC: Bastian Koppelmann <[e-mail]>
> CC: David Hildenbrand <[e-mail]>
> CC: Thomas Huth <[e-mail]>
> CC: Eric Farman <[e-mail]>
> CC: Matthew Rosato <[e-mail]>
> CC: Hannes Reinecke <[e-mail]>
> CC: Michael Walle <[e-mail]>
> CC: Artyom Tarasenko <[e-mail]>
> CC: Stefan Berger <[e-mail]>
> CC: Samuel Thibault <[e-mail]>
> CC: Alex Williamson <[e-mail]>
> CC: Tony Krowiak <[e-mail]>
> CC: Pierre Morel <[e-mail]>
> CC: Michael Roth <[e-mail]>
> CC: Hailiang Zhang <[e-mail]>
> CC: Juan Quintela <[e-mail]>
> CC: "Dr. David Alan Gilbert" <[e-mail]>
> CC: Luigi Rizzo <[e-mail]>
> CC: Giuseppe Lettieri <[e-mail]>
> CC: Vincenzo Maffione <[e-mail]>
> CC: Jan Kiszka <[e-mail]>
> CC: Anthony Green <[e-mail]>
> CC: Stafford Horne <[e-mail]>
> CC: Guan Xuetao <[e-mail]>
> CC: Max Filippov <[e-mail]>
> CC: [e-mail]
> CC: [e-mail]
> CC: [e-mail]
> CC: [e-mail]
> CC: [e-mail]
> CC: [e-mail]
> CC: [e-mail]
> CC: [e-mail]
>
>  python/commit-per-subsystem.py | 204 +++++++++++++++++++++++++++++++++
>  1 file changed, 204 insertions(+)
>  create mode 100755 python/commit-per-subsystem.py
>
> diff --git a/python/commit-per-subsystem.py b/python/commit-per-subsystem.
> py
> new file mode 100755
> index 0000000000..2ccf84cb15
> --- /dev/null
> +++ b/python/commit-per-subsystem.py
> @@ -0,0 +1,204 @@
> +#!/usr/bin/env python3
> +#
> +# Copyright (c) 2019 Virtuozzo International GmbH
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +import subprocess
> +import sys
> +import os
> +import glob
> +
> +
> +def git_add(pattern):
> +    subprocess.run(['git', 'add', pattern])
> +
> +
> +def git_commit(msg):
> +    subprocess.run(['git', 'commit', '-m', msg], capture_output=True)
> +
> +
> +def git_changed_files():
> +    ret = subprocess.check_output(['git', 'diff', '--name-only'],
> encoding='utf-8').split('\n')
> +    if ret[-1] == '':
> +        del ret[-1]
> +    return ret
> +
> +
> +maintainers = sys.argv[1]
> +message = sys.argv[2].strip()
> +
> +subsystem = None
> +
> +remap = {
> +    'Block layer core': 'block',
> +    'Block Jobs': 'block',
> +    'Dirty Bitmaps': 'block',
> +    'Block QAPI, monitor, command line': 'block',
> +    'Block I/O path': 'block',
> +    'Throttling infrastructure': 'block',
> +    'Architecture support': 's390x',
> +    'Guest CPU Cores (KVM)': 'kvm',
> +    'Guest CPU Cores (Xen)': 'xen',
> +    'Guest CPU cores (TCG)': 'tcg',
> +    'Network Block Device (NBD)': 'nbd',
> +    'Parallel NOR Flash devices': 'pflash',
> +    'Firmware configuration (fw_cfg)': 'fw_cfg',
> +    'Block SCSI subsystem': 'scsi',
> +    'Network device backends': 'net',
> +    'Netmap network backend': 'net',
> +    'Host Memory Backends': 'hostmem',
> +    'Cryptodev Backends': 'cryptodev',
> +    'QEMU Guest Agent': 'qga',
> +    'COLO Framework': 'colo',
> +    'Command line option argument parsing': 'cmdline',
> +    'Character device backends': 'chardev'
> +}
> +
> +
> +class Maintainers:
> +    def add(self, subsystem, path, mapper, mapper_name, glob_count=1):
> +        if subsystem in remap:
> +            subsystem = remap[subsystem]
> +        if subsystem not in self.subsystems:
> +            self.subsystems.append(subsystem)
> +
> +        if path[-1] == '/':
> +            path = path[:-1]
> +
> +        if path in mapper:
> +            if mapper[path][1] == glob_count:
> +                print('Warning: "{}" both in "{}" and "{}" in {} mapper
> with '
> +                      'same glob-count={}. {} ignored for this
> path.'.format(
> +                        path, mapper[path][0], subsystem, mapper_name,
> glob_count,
> +                          subsystem))
> +                return
> +            if mapper[path][1] < glob_count:
> +                # silently ignore worse match
> +                return
> +
> +        mapper[path] = (subsystem, glob_count)
> +
> +    def __init__(self, file_name):
> +        self.map_file = {}
> +        self.map_glob_file = {}
> +        self.map_dir = {}
> +        self.map_glob_dir = {}
> +        self.map_unmaintained_dir = {
> +            'python': ('python', 1),
> +            'hw/misc': ('misc', 1)
> +        }
> +        self.subsystems = ['python', 'misc']
> +        subsystem = None
> +
> +        with open(file_name) as f:
> +            mode2 = False
> +            prevline = ''
> +            for line in f:
> +                line = line.rstrip()
> +                if not line:
> +                    continue
> +                if len(line) >= 2 and line[1] == ':':
> +                    if line[0] == 'F':
> +                        fname = line[3:]
> +                        if fname in ['*', '*/']:
> +                            continue
> +                        if os.path.isfile(fname):
> +                            self.add(subsystem, fname, self.map_file,
> 'file')
> +                        elif os.path.isdir(fname):
> +                            self.add(subsystem, fname, self.map_dir,
> 'dir')
> +                        else:
> +                            paths = glob.glob(fname)
> +                            if not paths:
> +                                print('Warning: nothing corresponds to
> "{}"'.format(fname))
> +                                continue
> +
> +                            n = len(paths)
> +                            for f in paths:
> +                                if os.path.isfile(f):
> +                                    self.add(subsystem, f,
> self.map_glob_file, 'glob-file', n)
> +                                else:
> +                                    assert os.path.isdir(f)
> +                                    self.add(subsystem, f,
> self.map_glob_dir, 'glob-dir', n)
> +                elif line[:3] == '---':
> +                    subsystem = prevline
> +                    if subsystem == 'Devices':
> +                        mode2 = True
> +                elif mode2:
> +                    subsystem = line
> +                prevline = line
> +
> +    def find_in_map_dir(self, file_name, mapper):
> +        while file_name != '' and file_name not in mapper:
> +            file_name = os.path.dirname(file_name)
> +
> +        return None if file_name == '' else mapper[file_name][0]
> +
> +    def find_in_map_file(self, file_name, mapper):
> +        if file_name in mapper:
> +            return mapper[file_name][0]
> +
> +    def find_subsystem(self, file_name):
> +        s = self.find_in_map_file(file_name, self.map_file)
> +        if s is not None:
> +            return s
> +
> +        s = self.find_in_map_file(file_name, self.map_glob_file)
> +        if s is not None:
> +            return s
> +
> +        s = self.find_in_map_dir(file_name, self.map_dir)
> +        if s is not None:
> +            return s
> +
> +        s = self.find_in_map_dir(file_name, self.map_glob_dir)
> +        if s is not None:
> +            return s
> +
> +        s = self.find_in_map_dir(file_name, self.map_unmaintained_dir)
> +        if s is not None:
> +            return s
> +
> +        self.subsystems.append(file_name)
> +        return file_name
> +
> +
> +def commit(subsystem):
> +    msg = subsystem
> +    if msg in remap:
> +        msg = remap[msg]
> +    msg += ': ' + message
> +    git_commit(msg)
> +
> +mnt = Maintainers(maintainers)
> +res = {}
> +for f in git_changed_files():
> +    s = mnt.find_subsystem(f)
> +    if s in res:
> +        res[s].append(f)
> +    else:
> +        res[s] = [f]
> +
> +for s in mnt.subsystems:
> +    if s in res:
> +        print(s)
> +        for f in res[s]:
> +            print('  ', f)
> +
> +for s in mnt.subsystems:
> +    if s in res:
> +        for f in res[s]:
> +            git_add(f)
> +        commit(s)
> --
> 2.21.0
>
>
>

Cornelia Huck Nov. 12, 2019, 1:08 p.m. UTC | #3

On Fri, 11 Oct 2019 19:04:12 +0300
Vladimir Sementsov-Ogievskiy <[e-mail]> wrote:

> Add script to automatically commit tree-wide changes per-subsystem.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <[e-mail]>
> ---

I think this still needs some notes as to the supposed usage.

Patch

diff --git a/python/commit-per-subsystem.py b/python/commit-per-subsystem.py
new file mode 100755
index 0000000000..2ccf84cb15
--- /dev/null
+++ b/python/commit-per-subsystem.py
@@ -0,0 +1,204 @@ 
+#!/usr/bin/env python3
+#
+# Copyright (c) 2019 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import subprocess
+import sys
+import os
+import glob
+
+
+def git_add(pattern):
+    subprocess.run(['git', 'add', pattern])
+
+
+def git_commit(msg):
+    subprocess.run(['git', 'commit', '-m', msg], capture_output=True)
+
+
+def git_changed_files():
+    ret = subprocess.check_output(['git', 'diff', '--name-only'], encoding='utf-8').split('\n')
+    if ret[-1] == '':
+        del ret[-1]
+    return ret
+
+
+maintainers = sys.argv[1]
+message = sys.argv[2].strip()
+
+subsystem = None
+
+remap = {
+    'Block layer core': 'block',
+    'Block Jobs': 'block',
+    'Dirty Bitmaps': 'block',
+    'Block QAPI, monitor, command line': 'block',
+    'Block I/O path': 'block',
+    'Throttling infrastructure': 'block',
+    'Architecture support': 's390x',
+    'Guest CPU Cores (KVM)': 'kvm',
+    'Guest CPU Cores (Xen)': 'xen',
+    'Guest CPU cores (TCG)': 'tcg',
+    'Network Block Device (NBD)': 'nbd',
+    'Parallel NOR Flash devices': 'pflash',
+    'Firmware configuration (fw_cfg)': 'fw_cfg',
+    'Block SCSI subsystem': 'scsi',
+    'Network device backends': 'net',
+    'Netmap network backend': 'net',
+    'Host Memory Backends': 'hostmem',
+    'Cryptodev Backends': 'cryptodev',
+    'QEMU Guest Agent': 'qga',
+    'COLO Framework': 'colo',
+    'Command line option argument parsing': 'cmdline',
+    'Character device backends': 'chardev'
+}
+
+
+class Maintainers:
+    def add(self, subsystem, path, mapper, mapper_name, glob_count=1):
+        if subsystem in remap:
+            subsystem = remap[subsystem]
+        if subsystem not in self.subsystems:
+            self.subsystems.append(subsystem)
+
+        if path[-1] == '/':
+            path = path[:-1]
+
+        if path in mapper:
+            if mapper[path][1] == glob_count:
+                print('Warning: "{}" both in "{}" and "{}" in {} mapper with '
+                      'same glob-count={}. {} ignored for this path.'.format(
+                        path, mapper[path][0], subsystem, mapper_name, glob_count,
+                          subsystem))
+                return
+            if mapper[path][1] < glob_count:
+                # silently ignore worse match
+                return
+
+        mapper[path] = (subsystem, glob_count)
+
+    def __init__(self, file_name):
+        self.map_file = {}
+        self.map_glob_file = {}
+        self.map_dir = {}
+        self.map_glob_dir = {}
+        self.map_unmaintained_dir = {
+            'python': ('python', 1),
+            'hw/misc': ('misc', 1)
+        }
+        self.subsystems = ['python', 'misc']
+        subsystem = None
+
+        with open(file_name) as f:
+            mode2 = False
+            prevline = ''
+            for line in f:
+                line = line.rstrip()
+                if not line:
+                    continue
+                if len(line) >= 2 and line[1] == ':':
+                    if line[0] == 'F':
+                        fname = line[3:]
+                        if fname in ['*', '*/']:
+                            continue
+                        if os.path.isfile(fname):
+                            self.add(subsystem, fname, self.map_file, 'file')
+                        elif os.path.isdir(fname):
+                            self.add(subsystem, fname, self.map_dir, 'dir')
+                        else:
+                            paths = glob.glob(fname)
+                            if not paths:
+                                print('Warning: nothing corresponds to "{}"'.format(fname))
+                                continue
+
+                            n = len(paths)
+                            for f in paths:
+                                if os.path.isfile(f):
+                                    self.add(subsystem, f, self.map_glob_file, 'glob-file', n)
+                                else:
+                                    assert os.path.isdir(f)
+                                    self.add(subsystem, f, self.map_glob_dir, 'glob-dir', n)
+                elif line[:3] == '---':
+                    subsystem = prevline
+                    if subsystem == 'Devices':
+                        mode2 = True
+                elif mode2:
+                    subsystem = line
+                prevline = line
+
+    def find_in_map_dir(self, file_name, mapper):
+        while file_name != '' and file_name not in mapper:
+            file_name = os.path.dirname(file_name)
+
+        return None if file_name == '' else mapper[file_name][0]
+
+    def find_in_map_file(self, file_name, mapper):
+        if file_name in mapper:
+            return mapper[file_name][0]
+
+    def find_subsystem(self, file_name):
+        s = self.find_in_map_file(file_name, self.map_file)
+        if s is not None:
+            return s
+
+        s = self.find_in_map_file(file_name, self.map_glob_file)
+        if s is not None:
+            return s
+
+        s = self.find_in_map_dir(file_name, self.map_dir)
+        if s is not None:
+            return s
+
+        s = self.find_in_map_dir(file_name, self.map_glob_dir)
+        if s is not None:
+            return s
+
+        s = self.find_in_map_dir(file_name, self.map_unmaintained_dir)
+        if s is not None:
+            return s
+
+        self.subsystems.append(file_name)
+        return file_name
+
+
+def commit(subsystem):
+    msg = subsystem
+    if msg in remap:
+        msg = remap[msg]
+    msg += ': ' + message
+    git_commit(msg)
+
+mnt = Maintainers(maintainers)
+res = {}
+for f in git_changed_files():
+    s = mnt.find_subsystem(f)
+    if s in res:
+        res[s].append(f)
+    else:
+        res[s] = [f]
+
+for s in mnt.subsystems:
+    if s in res:
+        print(s)
+        for f in res[s]:
+            print('  ', f)
+
+for s in mnt.subsystems:
+    if s in res:
+        for f in res[s]:
+            git_add(f)
+        commit(s)