From patchwork Wed Mar 18 01:12:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 11444379 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7EA8D6CA for ; Wed, 18 Mar 2020 01:13:39 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5460C206EC for ; Wed, 18 Mar 2020 01:13:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fVutRdIO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5460C206EC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:43872 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jENH8-0000Cm-Gi for patchwork-qemu-devel@patchwork.kernel.org; Tue, 17 Mar 2020 21:13:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57374) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jENG0-0007Rt-Vm for qemu-devel@nongnu.org; Tue, 17 Mar 2020 21:12:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jENFz-0004UN-Gm for qemu-devel@nongnu.org; Tue, 17 Mar 2020 21:12:28 -0400 Received: from us-smtp-delivery-74.mimecast.com ([63.128.21.74]:32354) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jENFz-0004QY-Bc for qemu-devel@nongnu.org; Tue, 17 Mar 2020 21:12:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584493946; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yKMemzxXZBqk7hT78k1ip1hdWck5pBCC98Uw9VTFjSo=; b=fVutRdIOZH3e+9Si4RSxAOYsVR+dJ5JUsO131hN3s4/yZs3eOmTH8ninORn89JJUjJWwKQ RUkdQ1czrfXOmQuCHVEAmyO+Hw3zl1L7eELIwsNfVJ7COOS11jdbpUduzSlQVdTRZca8qm GZlg/2sKYoUzxp8FEqaCahxnm2GUkLM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-270-xYSISwthPhuw69n-uEXj1g-1; Tue, 17 Mar 2020 21:12:22 -0400 X-MC-Unique: xYSISwthPhuw69n-uEXj1g-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D591B1857BE0; Wed, 18 Mar 2020 01:12:21 +0000 (UTC) Received: from localhost (ovpn-118-148.rdu2.redhat.com [10.10.118.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id 731B37386D; Wed, 18 Mar 2020 01:12:21 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org, Cleber Rosa , Peter Maydell Subject: [PULL 1/4] scripts/simplebench: add simplebench.py Date: Tue, 17 Mar 2020 21:12:14 -0400 Message-Id: <20200318011217.2102748-2-ehabkost@redhat.com> In-Reply-To: <20200318011217.2102748-1-ehabkost@redhat.com> References: <20200318011217.2102748-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 63.128.21.74 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Vladimir Sementsov-Ogievskiy , Aleksandar Markovic Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Add simple benchmark table creator. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Aleksandar Markovic Message-Id: <20200228071914.11746-2-vsementsov@virtuozzo.com> Signed-off-by: Eduardo Habkost --- scripts/simplebench/simplebench.py | 128 +++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 scripts/simplebench/simplebench.py diff --git a/scripts/simplebench/simplebench.py b/scripts/simplebench/simplebench.py new file mode 100644 index 0000000000..59e7314ff6 --- /dev/null +++ b/scripts/simplebench/simplebench.py @@ -0,0 +1,128 @@ +#!/usr/bin/env python +# +# Simple benchmarking framework +# +# Copyright (c) 2019 Virtuozzo International GmbH. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + + +def bench_one(test_func, test_env, test_case, count=5, initial_run=True): + """Benchmark one test-case + + test_func -- benchmarking function with prototype + test_func(env, case), which takes test_env and test_case + arguments and returns {'seconds': int} (which is benchmark + result) on success and {'error': str} on error. Returned + dict may contain any other additional fields. + test_env -- test environment - opaque first argument for test_func + test_case -- test case - opaque second argument for test_func + count -- how many times to call test_func, to calculate average + initial_run -- do initial run of test_func, which don't get into result + + Returns dict with the following fields: + 'runs': list of test_func results + 'average': average seconds per run (exists only if at least one run + succeeded) + 'delta': maximum delta between test_func result and the average + (exists only if at least one run succeeded) + 'n-failed': number of failed runs (exists only if at least one run + failed) + """ + if initial_run: + print(' #initial run:') + print(' ', test_func(test_env, test_case)) + + runs = [] + for i in range(count): + print(' #run {}'.format(i+1)) + res = test_func(test_env, test_case) + print(' ', res) + runs.append(res) + + result = {'runs': runs} + + successed = [r for r in runs if ('seconds' in r)] + if successed: + avg = sum(r['seconds'] for r in successed) / len(successed) + result['average'] = avg + result['delta'] = max(abs(r['seconds'] - avg) for r in successed) + + if len(successed) < count: + result['n-failed'] = count - len(successed) + + return result + + +def ascii_one(result): + """Return ASCII representation of bench_one() returned dict.""" + if 'average' in result: + s = '{:.2f} +- {:.2f}'.format(result['average'], result['delta']) + if 'n-failed' in result: + s += '\n({} failed)'.format(result['n-failed']) + return s + else: + return 'FAILED' + + +def bench(test_func, test_envs, test_cases, *args, **vargs): + """Fill benchmark table + + test_func -- benchmarking function, see bench_one for description + test_envs -- list of test environments, see bench_one + test_cases -- list of test cases, see bench_one + args, vargs -- additional arguments for bench_one + + Returns dict with the following fields: + 'envs': test_envs + 'cases': test_cases + 'tab': filled 2D array, where cell [i][j] is bench_one result for + test_cases[i] for test_envs[j] (i.e., rows are test cases and + columns are test environments) + """ + tab = {} + results = { + 'envs': test_envs, + 'cases': test_cases, + 'tab': tab + } + n = 1 + n_tests = len(test_envs) * len(test_cases) + for env in test_envs: + for case in test_cases: + print('Testing {}/{}: {} :: {}'.format(n, n_tests, + env['id'], case['id'])) + if case['id'] not in tab: + tab[case['id']] = {} + tab[case['id']][env['id']] = bench_one(test_func, env, case, + *args, **vargs) + n += 1 + + print('Done') + return results + + +def ascii(results): + """Return ASCII representation of bench() returned dict.""" + from tabulate import tabulate + + tab = [[""] + [c['id'] for c in results['envs']]] + for case in results['cases']: + row = [case['id']] + for env in results['envs']: + row.append(ascii_one(results['tab'][case['id']][env['id']])) + tab.append(row) + + return tabulate(tab)