From patchwork Sat Feb 22 00:59:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 13986470 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0024486349 for ; Sat, 22 Feb 2025 00:59:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740185993; cv=none; b=r02McOKjIIh3eQ2bMgBuj/17xDdTmOxsb/ZiH85AqTZm3xN5mjxj3eLNnqwt48J8bEvl5C0hnaolw0zoVoTGDMHA6boY0sh8kt/HKyhdbumOx/A9Fon8WP25865OIWLr2FYdBchgj/qvwD413xPq0QFDXrXXDahuvVP9nSZkTuo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740185993; c=relaxed/simple; bh=d7xOpvN3PBwODDBpHrJVbVRIufIiM6HbJawrLGHNh7A=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=rWsPEHIakXKXTNC5TPM06doURpQmyNaTobcaRFpAcSS+ANdCWpFUAwbEFIgr41eNWJMK4uP2ishPYgwtMcdI3a8T2aln69qLD64nTW8JiRJouBdbyHIa3NZijXW5qcj6UDXQ3HFl7zqhnFKBObqChfQ3QCgHrkWTJu0ZH/dL5V4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--vipinsh.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=UAC1MzFX; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--vipinsh.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="UAC1MzFX" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2fc4fc93262so5887672a91.1 for ; Fri, 21 Feb 2025 16:59:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740185991; x=1740790791; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=S9ETAjvvalTxJAz5WZGNMYcUgNTOmlxvFzAQwoX+eOc=; b=UAC1MzFXplah8TqCOziEzctzOvRupNwghjAf8zFMbI7R6z2KBf2aPuFL4DNaf4dZXJ y3o+x2wfWJl2qcGzJAFtAjUMsr17FDwMu9L1ERNO0p/CHKI2q4hDfeJtwXk0OeWRsSBE onUil0aa+l5zbZIFo5uiGeEx/FOWFoj8WFC1WE09BmV/tz1gc0bIuFcUc1LUGIEGUgW4 vYTF297M5iL04oShYILX0V89zmEXyMaZah8mj0JwkRAU7g0zTc06RATpjLmj8Rmp+8ix vGDv9B0fpms9HSOcVD7JS57RCBvi82z1YnpU8ttbxWUO26M7fxA7XvW/Z8KHy7TYtTjN nWLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740185991; x=1740790791; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=S9ETAjvvalTxJAz5WZGNMYcUgNTOmlxvFzAQwoX+eOc=; b=ZffPmU/RrghRBf8UHQLgAX0YuqpaFPJsM9HTb4qvv3bJYBXd/mwFodnW+pKsyvRXr6 9zbOUHGNb0zW0duL3beRo4GZnbondW8x1ozC053q2fj/bXRXrxU6eZ/A0FVRCweL4bVC aAhSaXNWVyeDMcO+khn3pWDbBZAKsAOvE1mpRWqeKRNKygGvwK5TXkH5DxSrBLJQS8pK pMU/oKIsqmZeuH7LlKF1moPJDkcTfJPh1aQ2GIGJFfr/rsV2yxZsrAT2PmHhhRy9jNC0 yILyBWfz1vBjsvKSD0TwRDI9ZkFlRUXNbAXptsdkwZyZQ9j/emuVoBg7fe0+Pq4nU7CD OeCQ== X-Gm-Message-State: AOJu0Yxg4dTVqofWXluEQHN27H88mSnCvc/25CjbSk2YaKRigNofz9lx 11n1mMiCtFFiusOZ6L/nHKNp6NTON0dif8m5V9q+P6CVd1HU8pRgpziosZAhz2/e69TCMVXywT8 vvpi1Y9ls0oXj9DN4xwggdI6vWvyN6sK0yLEUbhGH4tbGPlR9ZSNc+odR1ppXDwdpIdEXa/wILl svJoQGvbEAI2AsJ1oyaqZrjOIheDWaLFEa+Q== X-Google-Smtp-Source: AGHT+IHCI1eCveOzHRO9wymEe+6WfTHE/ANfyN9OtDD+61Q0zEDS7M7t5/7XLTAP4RotsZhFw+AuUixUqPQf X-Received: from pjg5.prod.google.com ([2002:a17:90b:3f45:b0:2ef:d136:17fc]) (user=vipinsh job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:41:b0:2fa:228d:5b03 with SMTP id 98e67ed59e1d1-2fce78abcd9mr8111088a91.19.1740185991168; Fri, 21 Feb 2025 16:59:51 -0800 (PST) Date: Fri, 21 Feb 2025 16:59:30 -0800 In-Reply-To: <20250222005943.3348627-1-vipinsh@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250222005943.3348627-1-vipinsh@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250222005943.3348627-3-vipinsh@google.com> Subject: [PATCH 2/2] KVM: selftests: Create KVM selftest runner From: Vipin Sharma To: kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org, linux-arm-kernel@lists.infradead.org Cc: seanjc@google.com, pbonzini@redhat.com, anup@brainfault.org, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, maz@kernel.org, oliver.upton@linux.dev, Vipin Sharma Create KVM selftest runner to run selftests and provide various options for execution. Provide following features in the runner: 1. --timeout/-t: Max time each test should finish in before killing it. 2. --jobs/-j: Run these many tests in parallel. 3. --tests: Provide space separated path of tests to execute. 4. --test_dirs: Directories to search for test files and run them. 5. --output/-o: Create the folder with given name and dump output of each test in a hierarchical way. 6. Add summary at the end. Runner needs testcase files which are provided in the previous patch. Following are the examples to start the runner (cwd is tools/testing/selftests/kvm) - Basic run: python3 runner --test_dirs testcases - Run specific test python3 runner --tests ./testcases/dirty_log_perf_test/default.test - Run tests parallel python3 runner --test_dirs testcases -j 10 - Run 5 tests parallely at a time, with the timeout of 10 seconds and dump output in "result" directory python3 runner --test_dirs testcases -j 5 -t 10 --output result Sample output from the above command: python3_binary runner --test_dirs testcases -j 5 -t 10 --output result 2025-02-21 16:45:46,774 | 16809 | INFO | [Passed] testcases/guest_print_test/default.test 2025-02-21 16:45:47,040 | 16809 | INFO | [Passed] testcases/kvm_create_max_vcpus/default.test 2025-02-21 16:45:49,244 | 16809 | INFO | [Passed] testcases/dirty_log_perf_test/default.test ... 2025-02-21 16:46:07,225 | 16809 | INFO | [Passed] testcases/x86_64/pmu_event_filter_test/default.test 2025-02-21 16:46:08,020 | 16809 | INFO | [Passed] testcases/x86_64/vmx_preemption_timer_test/default.test 2025-02-21 16:46:09,734 | 16809 | INFO | [Timed out] testcases/x86_64/pmu_counters_test/default.test 2025-02-21 16:46:10,202 | 16809 | INFO | [Passed] testcases/hardware_disable_test/default.test 2025-02-21 16:46:10,203 | 16809 | INFO | Tests ran: 85 tests 2025-02-21 16:46:10,204 | 16809 | INFO | Passed: 61 2025-02-21 16:46:10,204 | 16809 | INFO | Failed: 4 2025-02-21 16:46:10,204 | 16809 | INFO | Skipped: 17 2025-02-21 16:46:10,204 | 16809 | INFO | Timed out: 3 2025-02-21 16:46:10,204 | 16809 | INFO | No run: 0 Output dumped in result directory $ tree result/ result/ ├── log └── testcases ├── access_tracking_perf_test │   └── default.test │   ├── stderr │   └── stdout ├── coalesced_io_test │   └── default.test │   ├── stderr │   └── stdout ... results/log file will have the status of each test like the one printed on console. Each stderr and stdout will have data based on the execution. Runner is implemented in python and needs at least 3.6 version. Signed-off-by: Vipin Sharma --- tools/testing/selftests/kvm/.gitignore | 1 + .../testing/selftests/kvm/runner/__main__.py | 96 +++++++++++++++++++ tools/testing/selftests/kvm/runner/command.py | 42 ++++++++ .../testing/selftests/kvm/runner/selftest.py | 49 ++++++++++ .../selftests/kvm/runner/test_runner.py | 40 ++++++++ 5 files changed, 228 insertions(+) create mode 100644 tools/testing/selftests/kvm/runner/__main__.py create mode 100644 tools/testing/selftests/kvm/runner/command.py create mode 100644 tools/testing/selftests/kvm/runner/selftest.py create mode 100644 tools/testing/selftests/kvm/runner/test_runner.py diff --git a/tools/testing/selftests/kvm/.gitignore b/tools/testing/selftests/kvm/.gitignore index 550b7c2b4a0c..a23fd4b2cb5f 100644 --- a/tools/testing/selftests/kvm/.gitignore +++ b/tools/testing/selftests/kvm/.gitignore @@ -11,3 +11,4 @@ !Makefile !Makefile.kvm !*.test +!*.py diff --git a/tools/testing/selftests/kvm/runner/__main__.py b/tools/testing/selftests/kvm/runner/__main__.py new file mode 100644 index 000000000000..008d862757f2 --- /dev/null +++ b/tools/testing/selftests/kvm/runner/__main__.py @@ -0,0 +1,96 @@ +# SPDX-License-Identifier: GPL-2.0 +import pathlib +import argparse +import platform +import logging +import os +import enum +import test_runner + + +def cli(): + parser = argparse.ArgumentParser( + prog="KVM Selftests Runner", + description="Run KVM selftests with different configurations", + formatter_class=argparse.RawTextHelpFormatter + ) + + parser.add_argument("--tests", + nargs="*", + default=[], + help="Test cases to run. Provide the space separated test case file paths") + + parser.add_argument("--test_dirs", + nargs="*", + default=[], + help="Run tests in the given directory and all its sub directories. Provide the space separated paths to add multiple directories.") + + parser.add_argument("-j", + "--jobs", + default=1, + type=int, + help="Number of parallel test runners to start") + + parser.add_argument("-t", + "--timeout", + default=120, + type=int, + help="How long to wait for a single test to finish before killing it") + + parser.add_argument("-o", + "--output", + nargs='?', + help="Output directory for test results.") + + return parser.parse_args() + + +def setup_logging(args): + output = args.output + if output == None: + logging.basicConfig(level=logging.INFO, + format="%(asctime)s | %(process)d | %(levelname)8s | %(message)s") + else: + logging_file = os.path.join(output, "log") + pathlib.Path(output).mkdir(parents=True, exist_ok=True) + logging.basicConfig(level=logging.INFO, + format="%(asctime)s | %(process)d | %(levelname)8s | %(message)s", + handlers=[ + logging.FileHandler(logging_file, mode='w'), + logging.StreamHandler() + ]) + + +def fetch_tests_from_dirs(scan_dirs, exclude_dirs): + test_files = [] + for scan_dir in scan_dirs: + for root, dirs, files in os.walk(scan_dir): + dirs[:] = [dir for dir in dirs if dir not in exclude_dirs] + for file in files: + test_files.append(os.path.join(root, file)) + return test_files + + +def fetch_test_files(args): + exclude_dirs = ["aarch64", "x86_64", "riscv", "s390x"] + # Don't exclude tests of the current platform + exclude_dirs.remove(platform.machine()) + + test_files = args.tests + test_files.extend(fetch_tests_from_dirs(args.test_dirs, exclude_dirs)) + # Remove duplicates + test_files = list(dict.fromkeys(test_files)) + return test_files + + +def main(): + args = cli() + setup_logging(args) + test_files = fetch_test_files(args) + tr = test_runner.TestRunner( + test_files, args.output, args.timeout, args.jobs) + tr.start() + + +if __name__ == "__main__": + main() diff --git a/tools/testing/selftests/kvm/runner/command.py b/tools/testing/selftests/kvm/runner/command.py new file mode 100644 index 000000000000..a58f16fe4542 --- /dev/null +++ b/tools/testing/selftests/kvm/runner/command.py @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: GPL-2.0 +import contextlib +import subprocess +import os +import pathlib + + +class Command: + """Executes a command + + Just execute a command. Dump output to the directory if provided. + + Returns the exit code of the command. + """ + + def __init__(self, command, timeout=None, output_dir=None): + self.command = command + self.timeout = timeout + self.output_dir = output_dir + + def __run(self, output=None, error=None): + proc = subprocess.run(self.command, stdout=output, + stderr=error, universal_newlines=True, + shell=True, timeout=self.timeout) + return proc.returncode + + def run(self): + if self.output_dir is not None: + pathlib.Path(self.output_dir).mkdir(parents=True, exist_ok=True) + + output = None + error = None + with contextlib.ExitStack() as stack: + if self.output_dir is not None: + output_path = os.path.join(self.output_dir, "stdout") + output = stack.enter_context( + open(output_path, encoding="utf-8", mode="w")) + + error_path = os.path.join(self.output_dir, "stderr") + error = stack.enter_context( + open(error_path, encoding="utf-8", mode="w")) + return self.__run(output, error) diff --git a/tools/testing/selftests/kvm/runner/selftest.py b/tools/testing/selftests/kvm/runner/selftest.py new file mode 100644 index 000000000000..cdf5d1085c08 --- /dev/null +++ b/tools/testing/selftests/kvm/runner/selftest.py @@ -0,0 +1,49 @@ +# SPDX-License-Identifier: GPL-2.0 +import subprocess +import command +import pathlib +import enum +import os +import logging + + +class SelftestStatus(str, enum.Enum): + PASSED = "Passed" + FAILED = "Failed" + SKIPPED = "Skipped" + TIMED_OUT = "Timed out" + NO_RUN = "No run" + + def __str__(self): + return str.__str__(self) + + +class Selftest: + """A single test. + + A test which can be run on its own. + """ + + def __init__(self, test_path, output_dir=None, timeout=None,): + test_command = pathlib.Path(test_path).read_text().strip() + if not test_command: + raise ValueError("Empty test command in " + test_path) + + if output_dir is not None: + output_dir = os.path.join(output_dir, test_path) + self.test_path = test_path + self.command = command.Command(test_command, timeout, output_dir) + self.status = SelftestStatus.NO_RUN + + def run(self): + try: + ret = self.command.run() + if ret == 0: + self.status = SelftestStatus.PASSED + elif ret == 4: + self.status = SelftestStatus.SKIPPED + else: + self.status = SelftestStatus.FAILED + except subprocess.TimeoutExpired as e: + # logging.error(type(e).__name__ + str(e)) + self.status = SelftestStatus.TIMED_OUT diff --git a/tools/testing/selftests/kvm/runner/test_runner.py b/tools/testing/selftests/kvm/runner/test_runner.py new file mode 100644 index 000000000000..b9d34c20bf88 --- /dev/null +++ b/tools/testing/selftests/kvm/runner/test_runner.py @@ -0,0 +1,40 @@ +# SPDX-License-Identifier: GPL-2.0 +import queue +import concurrent.futures +import logging +import time +import selftest + + +class TestRunner: + def __init__(self, test_files, output_dir, timeout, parallelism): + self.parallelism = parallelism + self.tests = [] + + for test_file in test_files: + self.tests.append(selftest.Selftest( + test_file, output_dir, timeout)) + + def _run(self, test): + test.run() + return test + + def start(self): + + status = {x: 0 for x in selftest.SelftestStatus} + count = 0 + with concurrent.futures.ProcessPoolExecutor(max_workers=self.parallelism) as executor: + all_futures = [] + for test in self.tests: + future = executor.submit(self._run, test) + all_futures.append(future) + + for future in concurrent.futures.as_completed(all_futures): + test = future.result() + logging.info(f"[{test.status}] {test.test_path}") + status[test.status] += 1 + count += 1 + + logging.info(f"Tests ran: {count} tests") + for result, count in status.items(): + logging.info(f"{result}: {count}")