From patchwork Wed Sep 5 13:49:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tvrtko Ursulin X-Patchwork-Id: 10588893 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 27E0B5A4 for ; Wed, 5 Sep 2018 13:50:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 15FDC2A21E for ; Wed, 5 Sep 2018 13:50:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 147252A242; Wed, 5 Sep 2018 13:50:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4C4CD2A21E for ; Wed, 5 Sep 2018 13:50:11 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 771FE6E4BF; Wed, 5 Sep 2018 13:50:07 +0000 (UTC) X-Original-To: Intel-gfx@lists.freedesktop.org Delivered-To: Intel-gfx@lists.freedesktop.org Received: from mail-wm0-x241.google.com (mail-wm0-x241.google.com [IPv6:2a00:1450:400c:c09::241]) by gabe.freedesktop.org (Postfix) with ESMTPS id 620476E4B6 for ; Wed, 5 Sep 2018 13:49:58 +0000 (UTC) Received: by mail-wm0-x241.google.com with SMTP id j192-v6so7871821wmj.1 for ; Wed, 05 Sep 2018 06:49:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Gkj/1+pNuPkYyVEUdmtBJsAE+lfWePKjxD3acbWhmCk=; b=jjENx4fMTAz4M4gYWa1vCYVAX0a+NewMdO8R8zT7yYqBz9/XNyHQ8nkch+zjIEmMcP DIvPFqUlfEM3F8TvRLFBq4WO/C7Zm+CLUaFSNHpvvDv7WtTTXoBaSz3kg4zBClDmZpQZ QMHvMSug5sCAHHZymHaPV/c000IcpBXct2j28zgwRbwIWVA7FWwwdVgr+W4OrYiSLljW w0N8xp2qIYVOqWDXkZQfTvxjmoEbAZQcCYOWeo2rMoAoe6Y7+FRLXRs+ZFclvuEW/6Al /Y6WNIqpKgz9YchTVNNwZcMGFDrD0EMAPljR4PirrnXdbb18I8q1sGIXp5Tw1uQHT5AV OOBA== X-Gm-Message-State: APzg51BFu5CCVqdYe+vrWoQmBAGgVgIrpPh0CpFS+QbOjml89AwVUT/w 5XjzszDYWFCDcBK7vVfTAxfM2w== X-Google-Smtp-Source: ANB0VdYwzO8SDf9B7/gyRzvm8aNRtJ2fqDWK43SgpM/40CvBGXBwQEdd7lomKPVpDGI+1uoYjckeuQ== X-Received: by 2002:a1c:e0d7:: with SMTP id x206-v6mr291504wmg.74.1536155396806; Wed, 05 Sep 2018 06:49:56 -0700 (PDT) Received: from localhost.localdomain ([95.144.165.37]) by smtp.gmail.com with ESMTPSA id j66-v6sm2701639wrj.28.2018.09.05.06.49.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Sep 2018 06:49:56 -0700 (PDT) From: Tvrtko Ursulin X-Google-Original-From: Tvrtko Ursulin To: igt-dev@lists.freedesktop.org Date: Wed, 5 Sep 2018 14:49:39 +0100 Message-Id: <20180905134939.2942-14-tvrtko.ursulin@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180905134939.2942-1-tvrtko.ursulin@linux.intel.com> References: <20180905134939.2942-1-tvrtko.ursulin@linux.intel.com> Subject: [Intel-gfx] [PATH i-g-t 13/13] media-bench: Add mixed mode evaluation X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Intel-gfx@lists.freedesktop.org MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Virus-Scanned: ClamAV using ClamSMTP From: Tvrtko Ursulin Mixed mode (-m) enables evaluation of different workload sets against one or more load balancing strategies. Contrary to the default mode which runs all selected workloads serialy, mixed mode runs a second stage where they are all run in parallel. The performance difference between the two passes is then used for the scoring metric. First metric is the normalized aggregate throughput, and second is balancer "fairness" as approximated by throughput achieved in mixed mode, relative to the best individual balancer for each workload. Signed-off-by: Tvrtko Ursulin --- scripts/media-bench.pl | 150 +++++++++++++++++++++++++++++++++-------- 1 file changed, 121 insertions(+), 29 deletions(-) diff --git a/scripts/media-bench.pl b/scripts/media-bench.pl index 77e75c78b113..066b542f95df 100755 --- a/scripts/media-bench.pl +++ b/scripts/media-bench.pl @@ -41,6 +41,8 @@ my $show_cmds = 0; my $realtime_target = 0; my $wps_target = 0; my $wps_target_param = 0; +my $multi_mode = 0; +my @multi_workloads; my $w_direct; my $balancer; my $nop; @@ -141,6 +143,7 @@ sub run_workload { my (@args) = @_; my ($time, $wps, $cmd); + my @ret; @args = add_wps_arg(@args); push @args, '-2' if $gt2; @@ -155,11 +158,13 @@ sub run_workload if (/^(\d+\.\d+)s elapsed \((\d+\.?\d+) workloads\/s\)$/) { $time = $1; $wps = $2; + } elsif (/(\d+)\: \d+\.\d+s elapsed \(\d+ cycles, (\d+\.?\d+) workloads\/s\)/) { + $ret[$1] = $2; } } close WSIM; - return ($time, $wps); + return ($time, $wps, \@ret); } sub dump_cmd @@ -223,7 +228,11 @@ sub trace_workload } close CMD; - $wrk =~ s/ /_/g; + $wrk =~ s/$wrk_root//g; + $wrk =~ s/\.wsim//g; + $wrk =~ s/-w/W/g; + $wrk =~ s/[ -]/_/g; + $wrk =~ s/\//-/g; $b =~ s/[ <>]/_/g; $file = "${wrk}_${b}_-r${r}_-c${c}"; @@ -259,6 +268,7 @@ sub calibrate_workload ($time, $wps) = run_workload(@args); + $wps = $r / $time if $w_direct; $error = abs($time - $client_target_s) / $client_target_s; last if $error <= $tol; @@ -278,13 +288,15 @@ sub calibrate_workload sub find_saturation_point { my ($wrk, $rr, $verbose, @args) = @_; - my ($last_wps, $c, $swps); + my ($last_wps, $c, $swps, $wwps); my $target = $realtime_target > 0 ? $realtime_target : $wps_target; my $r = $rr; my $wcnt; my $maxc; my $max = 0; + push @args, '-v' if $multi_mode and $w_direct; + if (defined $w_direct) { push @args, split /\s+/, $wrk; $wcnt = () = $wrk =~ /-[wW]/gi; @@ -297,8 +309,9 @@ sub find_saturation_point for ($c = 1; ; $c = $c + 1) { my ($time, $wps); + my @args_ = (@args, ('-r', $r, '-c', $c)); - ($time, $wps) = run_workload((@args, ('-r', $r, '-c', $c))); + ($time, $wps, $wwps) = run_workload(@args_); say " $c clients is $wps wps." if $verbose; @@ -324,21 +337,23 @@ sub find_saturation_point $r = int($rr * ($client_target_s / $time)); } elsif ($c == 1) { $swps = $wps; - return ($c, $wps, $swps) if $wcnt > 1 or - ($wps_target_param < 0 and $wps_target == 0); + return ($c, $wps, $swps, $wwps) if $wcnt > 1 or + $multi_mode or + ($wps_target_param < 0 and + $wps_target == 0); } $last_wps = $wps; } if ($target <= 0) { - return ($maxc, $max, $swps); + return ($maxc, $max, $swps, $wwps); } else { - return ($c - 1, $last_wps, $swps); + return ($c - 1, $last_wps, $swps, $wwps); } } -getopts('hv2xn:b:W:B:r:t:i:R:T:w:', \%opts); +getopts('hv2xmn:b:W:B:r:t:i:R:T:w:', \%opts); if (defined $opts{'h'}) { print < 0; say "Wps target is ${wps_target} wps." if $wps_target > 0; +say "Multi-workload mode." if $multi_mode; $nop = $opts{'n'}; $nop = calibrate_nop() unless $nop; say "Nop calibration is $nop."; goto VERIFY if defined $balancer; -my %best_bal; +my (%best_bal, %best_bid); my %results; my %scores; my %wscores; @@ -438,13 +464,22 @@ sub add_points } } -foreach my $wrk (@workloads) { +my @saturation_workloads = $multi_mode ? @multi_workloads : @workloads; +my %allwps; +my $widx = 0; + +push @saturation_workloads, '-w ' . join ' -w ', map("$wrk_root/$_", @workloads) + if $multi_mode; + +foreach my $wrk (@saturation_workloads) { my @args = ( "-n $nop"); my ($r, $error, $should_b, $best); my (%wps, %cwps, %mwps); my @sorted; my $range; + $w_direct = $wrk if $multi_mode and $widx == $#saturation_workloads; + $should_b = 1; $should_b = can_balance_workload($wrk) unless defined $w_direct; @@ -459,7 +494,7 @@ foreach my $wrk (@workloads) { GBAL: foreach my $G ('', '-G', '-d', '-G -d') { foreach my $H ('', '-H') { my @xargs; - my ($w, $c, $s); + my ($w, $c, $s, $bwwps); my $bid; if ($bal ne '') { @@ -476,21 +511,67 @@ foreach my $wrk (@workloads) { $wps_target = 0 if $wps_target_param < 0; - ($c, $w, $s) = find_saturation_point($wrk, $r, - 0, - (@args, - @xargs)); + ($c, $w, $s, $bwwps) = + find_saturation_point($wrk, $r, 0, + (@args, @xargs)); if ($wps_target_param < 0) { $wps_target = $s / -$wps_target_param; - ($c, $w, $s) = + ($c, $w, $s, $bwwps) = find_saturation_point($wrk, $r, 0, (@args, @xargs)); } + if ($multi_mode and $w_direct) { + my $widx; + + die unless scalar(@multi_workloads) == + scalar(@{$bwwps}); + die unless scalar(@multi_workloads) == + scalar(keys %allwps); + + # Total of all workload wps from the + # mixed run. + $w = 0; + foreach $widx (0..$#{$bwwps}) { + $w += $bwwps->[$widx]; + } + + # Total of all workload wps from when + # ran individually with the best + # balancer. + my $tot = 0; + foreach my $wrk (@multi_workloads) { + $tot += $allwps{$wrk}->{$best_bid{$wrk}}; + } + + # Normalize mixed sum with sum of + # individual runs. + $w *= 100; + $w /= $tot; + + # Second metric is average of each + # workload wps normalized by their + # individual run performance with the + # best balancer. + $s = 0; + $widx = 0; + foreach my $wrk (@multi_workloads) { + $s += 100 * $bwwps->[$widx] / + $allwps{$wrk}->{$best_bid{$wrk}}; + $widx++; + } + $s /= scalar(@multi_workloads); + + say sprintf('Aggregate (normalized) %.2f%%; fairness %.2f%%', + $w, $s); + } else { + $allwps{$wrk} = \%wps; + } + $wps{$bid} = $w; $cwps{$bid} = $s; @@ -500,7 +581,8 @@ foreach my $wrk (@workloads) { $mwps{$bid} = $w + $s; } - say "$c clients ($w wps, $s wps single client, score=$mwps{$bid})."; + say "$c clients ($w wps, $s wps single client, score=$mwps{$bid})." + unless $multi_mode and $w_direct; last BAL unless $should_b; next BAL if $bal eq ''; @@ -509,19 +591,24 @@ foreach my $wrk (@workloads) { } } + $widx++; + @sorted = sort { $mwps{$b} <=> $mwps{$a} } keys %mwps; - $best = $sorted[0]; + $best_bid{$wrk} = $sorted[0]; @sorted = sort { $b <=> $a } values %mwps; $range = 1 - $sorted[-1] / $sorted[0]; $best_bal{$wrk} = $sorted[0]; - say " Best balancer is '$best' (range=$range)."; + + next if $multi_mode and not $w_direct; + + say " Best balancer is '$best_bid{$wrk}' (range=$range)."; $results{$wrk} = \%mwps; add_points(\%wps, \%scores, \%wscores); - add_points(\%cwps, \%cscores, \%cwscores); add_points(\%mwps, \%mscores, \%mwscores); + add_points(\%cwps, \%cscores, \%cwscores); } sub dump_scoreboard @@ -552,12 +639,12 @@ sub dump_scoreboard return $balancer; } -dump_scoreboard('Total wps', \%scores); -dump_scoreboard('Total weighted wps', \%wscores); -dump_scoreboard('Per client wps', \%cscores); -dump_scoreboard('Per client weighted wps', \%cwscores); -dump_scoreboard('Combined wps', \%mscores); -$balancer = dump_scoreboard('Combined weighted wps', \%mwscores); +dump_scoreboard($multi_mode ? 'Throughput' : 'Total wps', \%scores); +dump_scoreboard('Total weighted wps', \%wscores) unless $multi_mode; +dump_scoreboard($multi_mode ? 'Fairness' : 'Per client wps', \%cscores); +dump_scoreboard('Per client weighted wps', \%cwscores) unless $multi_mode; +$balancer = dump_scoreboard($multi_mode ? 'Combined' : 'Combined wps', \%mscores); +$balancer = dump_scoreboard('Combined weighted wps', \%mwscores) unless $multi_mode; VERIFY: @@ -568,6 +655,11 @@ die unless defined $balancer; say "\nBalancer is '$balancer'."; say "Idleness tolerance is $idle_tolerance_pct%."; +if ($multi_mode) { + $w_direct = '-w ' . join ' -w ', map("$wrk_root/$_", @workloads); + @workloads = ($w_direct); +} + foreach my $wrk (@workloads) { my @args = ( "-n $nop" ); my ($r, $error, $c, $wps, $swps);